{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 使用Numpy构建神经网络\n",
    "\n",
    "本节将使用Python语言和Numpy库来构建神经网络模型，向读者展示神经网络的基本概念和工作过程。\n",
    "\n",
    "\n",
    "## 构建神经网络/深度学习模型的基本步骤\n",
    "\n",
    "如之前的介绍，应用于不同场景的深度学习模型具备一定的通用性，均可以从下述五个步骤来完成模型的构建和训练。\n",
    "- 数据处理：从本地文件或网络地址读取数据，并做预处理操作，如校验数据的正确性等。\n",
    "- 模型设计：完成网络结构的设计（模型要素1），相当于模型的假设空间，即模型能够表达的关系集合。\n",
    "- 训练配置：设定模型采用的寻解算法（模型要素2），即优化器，并指定计算资源。\n",
    "- 训练过程：循环调用训练过程，每轮均包括前向计算 、损失函数（优化目标，模型要素3）和后向传播这三个步骤。\n",
    "- 保存模型：将训练好的模型保存，以备预测时调用。\n",
    "\n",
    "下面使用Python编写预测波士顿房价的模型，一样遵循这样的五个步骤。\n",
    "正是由于这个建模和训练的过程存在通用性，即不同的模型仅仅在模型三要素上不同，而五个步骤中的其它部分保持一致，深度学习框架才有用武之地。\n",
    "\n",
    "\n",
    "## 波士顿房价预测\n",
    "\n",
    "波士顿房价预测是一个经典的机器学习问题，类似于程序员世界的“Hello World”。波士顿地区的房价是由诸多因素影响的，该数据集统计了13种可能影响房价的因素和该类型房屋的均价，期望构建一个基于13个因素预测房价的模型。预测问题根据预测输出的类型是连续的实数值，还是离散的标签，区分为回归任务和分类任务。因为房价是一个连续值，所以房价预测显然是一个回归任务。下面我们尝试用最简单的线性回归模型解决这个问题，并用神经网络来实现这个模型。\n",
    "\n",
    "## 线性回归模型\n",
    "\n",
    "假设房价和各影响因素之间能够用线性关系来描述（类似牛顿第二定律的案例）：\n",
    "\n",
    "$$y = {\\sum_{j=1}^Mx_j w_j} + b$$\n",
    "\n",
    "模型的求解即是通过数据拟合出每个$w_j$和$b$。$w_j$和$b$分别表示该线性模型的权重和偏置。一维情况下，$w_j$和$b$就是直线的斜率和截距。\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 数据处理\n",
    "\n",
    "在搭建模型之前，让我们先导入数据，查阅下内容。房价数据存放在本地目录下的housing.data文件中，通过执行如下的代码可以导入数据并查阅。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([6.320e-03, 1.800e+01, 2.310e+00, ..., 3.969e+02, 7.880e+00,\n",
       "       1.190e+01])"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 导入需要用到的package\n",
    "import numpy as np\n",
    "import json\n",
    "# 读入训练数据\n",
    "datafile = './work/housing.data'\n",
    "data = np.fromfile(datafile, sep=' ')\n",
    "data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "因为读入的原始数据是1维的，所有数据都连在了一起。所以将数据的形状进行变换，形成一个2维的矩阵。每行为一个数据样本（14个值），每个数据样本包含13个X（影响房价的特征）和一个Y（该类型房屋的均价）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 读入之后的数据被转化成1维array，其中array的\n",
    "# 第0-13项是第一条数据，第14-27项是第二条数据，.... \n",
    "# 这里对原始数据做reshape，变成N x 14的形式\n",
    "feature_names = [ 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE','DIS', \n",
    "                 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV' ]\n",
    "feature_num = len(feature_names)\n",
    "data = data.reshape([data.shape[0] // feature_num, feature_num])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(14,)\n",
      "[6.320e-03 1.800e+01 2.310e+00 0.000e+00 5.380e-01 6.575e+00 6.520e+01\n",
      " 4.090e+00 1.000e+00 2.960e+02 1.530e+01 3.969e+02 4.980e+00 2.400e+01]\n"
     ]
    }
   ],
   "source": [
    "# 查看数据\n",
    "x = data[0]\n",
    "print(x.shape)\n",
    "print(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "取80%的数据作为训练集，预留20%的数据用于测试模型的预测效果（训练好的模型预测值与实际房价的差距）。打印训练集的形状可见，我们共有404个样本，每个样本含有13个特征和1个预测值。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(404, 14)"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ratio = 0.8\n",
    "offset = int(data.shape[0] * ratio)\n",
    "training_data = data[:offset]\n",
    "training_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "对每个特征进行归一化处理，使得每个特征的取值缩放到0~1之间。这样做有两个好处：\n",
    "1. 模型训练更高效。\n",
    "2. 特征前的权重大小可代表该变量对预测结果的贡献度（因为每个特征值本身的范围相同）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 计算train数据集的最大值，最小值，平均值\n",
    "maximums, minimums, avgs = \\\n",
    "                     training_data.max(axis=0), \\\n",
    "                     training_data.min(axis=0), \\\n",
    "     training_data.sum(axis=0) / training_data.shape[0]\n",
    "# 对数据进行归一化处理\n",
    "for i in range(feature_num):\n",
    "    #print(maximums[i], minimums[i], avgs[i])\n",
    "    data[:, i] = (data[:, i] - avgs[i]) / (maximums[i] - minimums[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "将上述几个数据处理操作合并成load data函数，并确认函数的执行效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "def load_data():\n",
    "    # 从文件导入数据\n",
    "    datafile = './work/housing.data'\n",
    "    data = np.fromfile(datafile, sep=' ')\n",
    "\n",
    "    # 每条数据包括14项，其中前面13项是影响因素，第14项是相应的房屋价格中位数\n",
    "    feature_names = [ 'CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', \\\n",
    "                      'DIS', 'RAD', 'TAX', 'PTRATIO', 'B', 'LSTAT', 'MEDV' ]\n",
    "    feature_num = len(feature_names)\n",
    "\n",
    "    # 将原始数据进行Reshape，变成[N, 14]这样的形状\n",
    "    data = data.reshape([data.shape[0] // feature_num, feature_num])\n",
    "\n",
    "    # 将原数据集拆分成训练集和测试集\n",
    "    # 这里使用80%的数据做训练，20%的数据做测试\n",
    "    # 测试集和训练集必须是没有交集的\n",
    "    ratio = 0.8\n",
    "    offset = int(data.shape[0] * ratio)\n",
    "    training_data = data[:offset]\n",
    "\n",
    "    # 计算train数据集的最大值，最小值，平均值\n",
    "    maximums, minimums, avgs = training_data.max(axis=0), training_data.min(axis=0), \\\n",
    "                                 training_data.sum(axis=0) / training_data.shape[0]\n",
    "\n",
    "    # 对数据进行归一化处理\n",
    "    for i in range(feature_num):\n",
    "        #print(maximums[i], minimums[i], avgs[i])\n",
    "        data[:, i] = (data[:, i] - avgs[i]) / (maximums[i] - minimums[i])\n",
    "\n",
    "    # 训练集和测试集的划分比例\n",
    "    training_data = data[:offset]\n",
    "    test_data = data[offset:]\n",
    "    return training_data, test_data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 获取数据\n",
    "training_data, test_data = load_data()\n",
    "x = training_data[:, :-1]\n",
    "y = training_data[:, -1:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.02146321  0.03767327 -0.28552309 -0.08663366  0.01289726  0.04634817\n",
      "  0.00795597 -0.00765794 -0.25172191 -0.11881188 -0.29002528  0.0519112\n",
      " -0.17590923]\n",
      "[-0.00390539]\n"
     ]
    }
   ],
   "source": [
    "# 查看数据\n",
    "print(x[0])\n",
    "print(y[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "如果将输入特征和输出预测值均以向量表示，输入特征x一共有13个分量，y只有1个分量，所以参数权重的形状（shape）应该是$13\\times1$。假设我们以如下任意数字赋值参数做初始化：\n",
    "$w=[0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, -0.1, -0.2, -0.3，-0.4, 0.0]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "w = [0.1, 0.2, 0.3, 0.4, 0.5, 0.6, 0.7, 0.8, -0.1, -0.2, -0.3, -0.4, 0.0]\n",
    "w = np.array(w).reshape([13, 1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "取出第1条样本数据，观察样本的特征向量与参数向量相乘之后的结果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.03395597]\n"
     ]
    }
   ],
   "source": [
    "x1=x[0]\n",
    "t = np.dot(x1, w)\n",
    "print(t)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "此外，完整的线性回归公式，还需要初始化偏移量$b$，同样随意赋初值-0.2。\n",
    "那么，线性回归模型的完整输出是$z=t+b$，这个从特征和参数计算输出值的过程称为“前向计算”。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.16604403]\n"
     ]
    }
   ],
   "source": [
    "b = -0.2\n",
    "z = t + b\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 构建神经网络\n",
    "\n",
    "将上述计算预测输出的过程以“类和对象”的方式来描述，实现的方案如下所示。类成员变量有参数 w 和 b，并写了一个forward函数（代表“前向计算”）完成上述从特征和参数到输出预测值的计算过程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，\n",
    "        # 此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "基于Network类的定义，模型的计算过程可以按下述方式达成。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[-0.63182506]\n"
     ]
    }
   ],
   "source": [
    "net = Network(13)\n",
    "x1 = x[0]\n",
    "y1 = y[0]\n",
    "z = net.forward(x1)\n",
    "print(z)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "通过模型计算$x_1$表示的影响因素所对应的房价应该是$z$, 但实际数据告诉我们房价是$y$，这时我们需要有某种指标来衡量预测值$z$跟真实值$y$之间的差距。对于回归问题，最常采用的衡量方法是使用均方误差作为评价模型好坏的指标，具体定义如下：\n",
    "$$Loss = (y - z)^2$$\n",
    "上式中的$Loss$(简记为: $L$) 通常也被称作损失函数，它是衡量模型好坏的指标，在回归问题中均方误差是一种比较常见的形式，分类问题中通常会采用交叉熵损失函数，在后续的章节中会更详细的介绍。\n",
    "对一个样本计算损失的代码实现如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[0.39428312]\n"
     ]
    }
   ],
   "source": [
    "Loss = (y1 - z)*(y1 - z)\n",
    "print(Loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "因为计算损失时需要把每个样本的损失都考虑到，所以我们需要对单个样本的损失函数进行求和，并除以样本总数$N$。\n",
    "$$L= \\frac{1}{N}\\sum_i{(y^{(i)} - z^{(i)})^2}$$\n",
    "对上面的计算代码做出相应的调整，在Network类下面添加损失函数的计算过程如下"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        cost = error * error\n",
    "        cost = np.mean(cost)\n",
    "        return cost\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "使用上面定义的Network类，可以方便的计算预测值和损失函数。\n",
    "需要注意，类中的变量x, w，b, z, error等均是向量。以变量x为例，共有两个维度，一个代表特征数量（=13），一个代表样本数量（演示程序如下）。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "predict:  [[-0.63182506]\n",
      " [-0.55793096]\n",
      " [-1.00062009]]\n",
      "loss: 0.7229825055441156\n"
     ]
    }
   ],
   "source": [
    "net = Network(13)\n",
    "# 此处可以一次性计算多个样本的预测值和损失函数\n",
    "x1 = x[0:3]\n",
    "y1 = y[0:3]\n",
    "z = net.forward(x1)\n",
    "print('predict: ', z)\n",
    "loss = net.loss(z, y1)\n",
    "print('loss:', loss)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 神经网络的训练\n",
    "\n",
    "上述计算过程描述了如何构建神经网络，通过神经网络完成预测值和损失函数的计算。接下来将介绍如何求解参数$w$和$b$的数值，这个过程也称为模型训练。模型训练的目标是让定义的损失函数尽可能的小，也就是说找到一个参数解$w$和$b$使得损失函数取得极小值。\n",
    "\n",
    "### 求解损失函数的极小值\n",
    "\n",
    "基于最基本的微积分知识，函数在极值点处的导数为0。那么，让损失函数取极小值的$w$和$b$应该是下述方程组的解：\n",
    "$$\\frac{\\partial{L}}{\\partial{w_j}}=0,  \\ \\ for \\ \\ \\ j = 0, ..., 12$$\n",
    "$$\\frac{\\partial{L}}{\\partial{b}}=0$$\n",
    "\n",
    "将样本数据$(x, y)$带入上面的方程组固然可以求解出$w$和$b$的值，但是这种方法只对线性回归这样简单的情况有效。如果模型中含有非线性变换，或者损失函数不是均方差这种简单形式，则很难通过上式求解。为了避免这一情况，下面我们将引入更加普适的数值求解方法。\n",
    "\n",
    "### 梯度下降法\n",
    "\n",
    "训练的关键是找到一组$(w, b)$使得损失函数$L$取极小值。我们先看一下损失函数$L$只随两个参数变化时的简单情形，启发下寻解的思路。\n",
    "$$L=L(w_5, w_9)$$\n",
    "这里我们将$w_0, w_1, ..., w_{12}$中除$w_5, w_9$之外的参数和$b$都固定下来，可以用图画出$L(w_5, w_9)$的形式。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "net = Network(13)\n",
    "losses = []\n",
    "#只画出参数w5和w9在区间[-160, 160]的曲线部分，已经包含损失函数的极值\n",
    "w5 = np.arange(-160.0, 160.0, 1.0)\n",
    "w9 = np.arange(-160.0, 160.0, 1.0)\n",
    "losses = np.zeros([len(w5), len(w9)])\n",
    "\n",
    "#计算设定区域内每个参数取值所对应的Loss\n",
    "for i in range(len(w5)):\n",
    "    for j in range(len(w9)):\n",
    "        net.w[5] = w5[i]\n",
    "        net.w[9] = w9[j]\n",
    "        z = net.forward(x)\n",
    "        loss = net.loss(z, y)\n",
    "        losses[i, j] = loss\n",
    "\n",
    "#将两个变量和对应的Loss作3D图\n",
    "import matplotlib.pyplot as plt\n",
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "fig = plt.figure()\n",
    "ax = Axes3D(fig)\n",
    "\n",
    "w5, w9 = np.meshgrid(w5, w9)\n",
    "\n",
    "ax.plot_surface(w5, w9, losses, rstride=1, cstride=1, cmap='rainbow')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 简单情形——只考虑两个参数$w_5$和$w_9$\n",
    "\n",
    "对于这种简单情形，我们利用上面的程序在3维空间中画出了损失函数随参数变化的曲面图，从上图可以看出有些区域的函数值明显比周围的点小。需要说明的是：为什么这里我们选择$w_5$和$w_9$来画图？这是因为选择这两个参数的时候，可比较直观的从损失函数的曲面图上发现极值点的存在。其他参数组合，从图形上观测损失函数的极值点不够直观。\n",
    "\n",
    "上文提到，直接求解导数方程的方式在多数情况下较困难，本质原因是导数方程往往正向求解容易（已知X，求得Y），反向求解较难（已知Y，求得X）。这种特性的方程在很多加密算法中较为常见，与日常见到的锁头特性一样：已知“钥匙”，锁头判断是否正确容易；已知“锁头”，反推钥匙的形状比较难。\n",
    "\n",
    "这种情况特别类似于一位想从山峰走到坡谷的盲人，他看不见坡谷在哪（无法逆向求解出Loss导数为0时的参数值），但可以伸脚探索身边的坡度（当前点的导数值，也称为梯度）。那么，求解Loss函数最小值可以“从当前的参数取值，一步步的按照下坡的方向下降，直到走到最低点”实现。这种方法个人称它为“瞎子下坡法”。哦不，有个更正式的说法“梯度下降法”。\n",
    "\n",
    "现在我们要找出一组$[w_5, w_9]$的值，使得损失函数最小，实现梯度下降法的方案如下：\n",
    "\n",
    "- 随机的选一组初始值，例如：\n",
    "    $[w_5, w_9] = [-100.0, -100.0]$\n",
    "- 选取下一个点$[w_5^{'} , w_9^{'}]$使得\n",
    "   $L(w_5^{'} , w_9^{'}) < L(w_5, w_9)$\n",
    "- 重复上面的步骤2，直到损失函数几乎不再下降\n",
    "\n",
    "\n",
    "![](https://ai-studio-static-online.cdn.bcebos.com/b8e37d9e937d41c89c34af3884ad05947d5a9ec02316467da1b6012f9b0635c4)\n",
    "\n",
    "图1-2-1 ：梯度下降方向示意图\n",
    "\n",
    "如何选择$[w_5^{'} , w_9^{'}]$是至关重要的，第一要保证$L$是下降的，第二要使得下降的趋势尽可能的快。微积分的基础知识告诉我们，沿着梯度的反方向，是函数值下降最快的方向，如下图所示在点$P_0$，$[w_5, w_9] = [-100.0, -100.0]$，梯度方向是图中$P_0$点的箭头指向的方向，沿着箭头方向向前移动一小步，可以观察损失函数的变化。\n",
    "在$P_0$点，$[w_5, w_9] = [-150.0, -150.0]$，可以计算出，此时的loss在1300左右。\n",
    "\n",
    "### 计算梯度\n",
    "\n",
    "上面我们讲过了损失函数的计算方法，这里稍微加以改写，引入因子$\\frac{1}{2}$，定义损失函数如下\n",
    "$$L= \\frac{1}{2N}\\sum_{i=1}^N{(y^{(i)} - z^{(i)})^2}$$\n",
    "其中$z_i$是网络对第$i$个样本的预测值\n",
    "$$z^{(i)} = \\sum_{j=0}^{12}{x_j^{(i)} w^{(j)}} + b$$\n",
    "\n",
    "可以计算出$L$对$w$和$b$的偏导数\n",
    "\n",
    "$$\\frac{\\partial{L}}{\\partial{w_j}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})\\frac{\\partial{z^{(i)}}}{w_j}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})x_j^{(i)}}$$\n",
    "\n",
    "$$\\frac{\\partial{L}}{\\partial{b}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})\\frac{\\partial{z^{(i)}}}{b}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})}$$\n",
    "\n",
    "从导数的计算过程可以看出，因子$\\frac{1}{2}$被消掉了，这是因为二次函数求导的时候会产生因子$2$，这也是我们将损失函数改写的原因\n",
    "\n",
    "这里我们感兴趣的是$w_5$和$w_9$，\n",
    "$$\\frac{\\partial{C}}{\\partial{w_5}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})x_5^{(i)}}$$\n",
    "$$\\frac{\\partial{C}}{\\partial{w_9}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})x_9^{(i)}}$$\n",
    "\n",
    "则可以在Network类中定义如下的梯度计算函数"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "梯度计算公式\n",
    "$$\\frac{\\partial{L}}{\\partial{w_j}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})\\frac{\\partial{z_j^{(i)}}}{w_j}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})x_j^{(i)}}$$\n",
    "\n",
    "借助于numpy里面的矩阵操作，我们可以直接对所有$w_j \\  (j = 0, ..., 12)$ 一次性的计算出13个参数所对应的梯度来\n",
    "\n",
    "先考虑只有一个样本的情况，上式中的$N=1$，$\\frac{\\partial{L}}{\\partial{w_j}}=(z^{(1)} - y^{(1)})x_j^{(1)}$\n",
    "\n",
    "可以通过具体的程序查看每个变量的数据和维度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x1 [-0.02146321  0.03767327 -0.28552309 -0.08663366  0.01289726  0.04634817\n",
      "  0.00795597 -0.00765794 -0.25172191 -0.11881188 -0.29002528  0.0519112\n",
      " -0.17590923], shape (13,)\n",
      "y1 [-0.00390539], shape (1,)\n",
      "z1 [-12.05947643], shape (1,)\n"
     ]
    }
   ],
   "source": [
    "x1 = x[0]\n",
    "y1 = y[0]\n",
    "z1 = net.forward(x1)\n",
    "print('x1 {}, shape {}'.format(x1, x1.shape))\n",
    "print('y1 {}, shape {}'.format(y1, y1.shape))\n",
    "print('z1 {}, shape {}'.format(z1, z1.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "按上面的公式，当只有一个样本时，可以计算某个$w_j$，比如$w_0$的梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w0 [0.25875126]\n"
     ]
    }
   ],
   "source": [
    "gradient_w0 = (z1 - y1) * x1[0]\n",
    "print('gradient_w0 {}'.format(gradient_w0))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "同样我们可以计算$w_1$的梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w1 [-0.45417275]\n"
     ]
    }
   ],
   "source": [
    "gradient_w1 = (z1 - y1) * x1[1]\n",
    "print('gradient_w1 {}'.format(gradient_w1))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "依次计算$w_2$的梯度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w1 [3.44214394]\n"
     ]
    }
   ],
   "source": [
    "gradient_w2= (z1 - y1) * x1[2]\n",
    "print('gradient_w1 {}'.format(gradient_w2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "聪明的读者可能已经想到，写一个for循环即可计算从$w_0$到$w_{12}$的所有权重的梯度，这留作作业题。\n",
    "\n",
    "Numpy给我们提供了更简单的操作方法，即使用矩阵操作。计算梯度的代码中直接用  (z1 - y1) * x1，得到的是一个13维的向量，每个分量分别代表该维度的梯度。Numpy的广播功能（对向量和矩阵计算如同对1个单一变量计算一样）是我们使用它的原因。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w_by_sample1 [ 0.25875126 -0.45417275  3.44214394  1.04441828 -0.15548386 -0.55875363\n",
      " -0.09591377  0.09232085  3.03465138  1.43234507  3.49642036 -0.62581917\n",
      "  2.12068622], gradient.shape (13,)\n"
     ]
    }
   ],
   "source": [
    "gradient_w = (z1 - y1) * x1\n",
    "print('gradient_w_by_sample1 {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "再回到上面的梯度计算公式\n",
    "$$\\frac{\\partial{L}}{\\partial{w_j}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})\\frac{\\partial{z^{(i)}}}{w_j}} = \\frac{1}{N}\\sum_i^N{(z^{(i)} - y^{(i)})x_j^{(i)}}$$\n",
    "\n",
    "这里输入数据中有多个样本，每个样本都对梯度有贡献。如上代码计算了只有样本1时的梯度值，同样的计算方法也可以计算样本2和样本3对梯度的贡献。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w_by_sample2 [ 0.7329239   4.91417754  3.33394253  2.9912385   4.45673435 -0.58146277\n",
      " -5.14623287 -2.4894594   7.19011988  7.99471607  0.83100061 -1.79236081\n",
      "  2.11028056], gradient.shape (13,)\n"
     ]
    }
   ],
   "source": [
    "x2 = x[1]\n",
    "y2 = y[1]\n",
    "z2 = net.forward(x2)\n",
    "gradient_w = (z2 - y2) * x2\n",
    "print('gradient_w_by_sample2 {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w_by_sample3 [ 0.25138584  1.68549775  1.14349809  1.02595515  1.5286008  -1.93302947\n",
      "  0.4058236  -0.85385157  2.46611579  2.74208162  0.28502219 -0.46695229\n",
      "  2.39363651], gradient.shape (13,)\n"
     ]
    }
   ],
   "source": [
    "x3 = x[2]\n",
    "y3 = y[2]\n",
    "z3 = net.forward(x3)\n",
    "gradient_w = (z3 - y3) * x3\n",
    "print('gradient_w_by_sample3 {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "可能有的读者再次想到可以使用for循环把每个样本对梯度的贡献都计算出来，然后再作平均。\n",
    "\n",
    "但是我们不需要这么做，仍然可以使用Numpy的矩阵操作来简化运算，比如三个样本的情况。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "x [[-0.02146321  0.03767327 -0.28552309 -0.08663366  0.01289726  0.04634817\n",
      "   0.00795597 -0.00765794 -0.25172191 -0.11881188 -0.29002528  0.0519112\n",
      "  -0.17590923]\n",
      " [-0.02122729 -0.14232673 -0.09655922 -0.08663366 -0.12907805  0.0168406\n",
      "   0.14904763  0.0721009  -0.20824365 -0.23154675 -0.02406783  0.0519112\n",
      "  -0.06111894]\n",
      " [-0.02122751 -0.14232673 -0.09655922 -0.08663366 -0.12907805  0.1632288\n",
      "  -0.03426854  0.0721009  -0.20824365 -0.23154675 -0.02406783  0.03943037\n",
      "  -0.20212336]], shape (3, 13)\n",
      "y [[-0.00390539]\n",
      " [-0.05723872]\n",
      " [ 0.23387239]], shape (3, 1)\n",
      "z [[-12.05947643]\n",
      " [-34.58467747]\n",
      " [-11.60858134]], shape (3, 1)\n"
     ]
    }
   ],
   "source": [
    "# 注意这里是一次取出3个样本的数据，不是取出第3个样本\n",
    "x3samples = x[0:3]\n",
    "y3samples = y[0:3]\n",
    "z3samples = net.forward(x3samples)\n",
    "\n",
    "print('x {}, shape {}'.format(x3samples, x3samples.shape))\n",
    "print('y {}, shape {}'.format(y3samples, y3samples.shape))\n",
    "print('z {}, shape {}'.format(z3samples, z3samples.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上面的x3samples, y3samples, z3samples的第一维大小均为3，表示有3个样本。下面计算这3个样本对梯度的贡献。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w [[ 0.25875126 -0.45417275  3.44214394  1.04441828 -0.15548386 -0.55875363\n",
      "  -0.09591377  0.09232085  3.03465138  1.43234507  3.49642036 -0.62581917\n",
      "   2.12068622]\n",
      " [ 0.7329239   4.91417754  3.33394253  2.9912385   4.45673435 -0.58146277\n",
      "  -5.14623287 -2.4894594   7.19011988  7.99471607  0.83100061 -1.79236081\n",
      "   2.11028056]\n",
      " [ 0.25138584  1.68549775  1.14349809  1.02595515  1.5286008  -1.93302947\n",
      "   0.4058236  -0.85385157  2.46611579  2.74208162  0.28502219 -0.46695229\n",
      "   2.39363651]], gradient.shape (3, 13)\n"
     ]
    }
   ],
   "source": [
    "gradient_w = (z3samples - y3samples) * x3samples\n",
    "print('gradient_w {}, gradient.shape {}'.format(gradient_w, gradient_w.shape))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "此处可见，计算梯度gradient_w的维度是$3 \\times 13$，并且其第1行与上面第1个样本计算的梯度gradient_w_by_sample1一致，第2行与上面第2个样本计算的梯度gradient_w_by_sample1一致，第3行与上面第3个样本计算的梯度gradient_w_by_sample1一致。这里使用矩阵操作，可能更加方便的对3个样本分别计算各自对梯度的贡献。\n",
    "\n",
    "那么对于有N个样本的情形，我们可以直接使用如下方式计算出所有样本对梯度的贡献，这就是使用Numpy库广播功能带来的便捷。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w shape (404, 13)\n",
      "[[  0.25875126  -0.45417275   3.44214394 ...   3.49642036  -0.62581917\n",
      "    2.12068622]\n",
      " [  0.7329239    4.91417754   3.33394253 ...   0.83100061  -1.79236081\n",
      "    2.11028056]\n",
      " [  0.25138584   1.68549775   1.14349809 ...   0.28502219  -0.46695229\n",
      "    2.39363651]\n",
      " ...\n",
      " [ 14.70025543 -15.10890735  36.23258734 ...  24.54882966   5.51071122\n",
      "   26.26098922]\n",
      " [  9.29832217 -15.33146159  36.76629344 ...  24.91043398  -1.27564923\n",
      "   26.61808955]\n",
      " [ 19.55115919 -10.8177237   25.94192351 ...  17.5765494    3.94557661\n",
      "   17.64891012]]\n"
     ]
    }
   ],
   "source": [
    "z = net.forward(x)\n",
    "gradient_w = (z - y) * x\n",
    "print('gradient_w shape {}'.format(gradient_w.shape))\n",
    "print(gradient_w)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上面gradient_w的每一行代表了一个样本对梯度的贡献。根据梯度的计算公式，总梯度是对每个样本对梯度贡献的平均值。\n",
    "我们也可以使用Numpy的均值函数来完成此过程："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w  (13,)\n",
      "w  (13, 1)\n",
      "[ 1.59697064 -0.92928123  4.72726926  1.65712204  4.96176389  1.18068454\n",
      "  4.55846519 -3.37770889  9.57465893 10.29870662  1.3900257  -0.30152215\n",
      "  1.09276043]\n",
      "[[ 1.76405235e+00]\n",
      " [ 4.00157208e-01]\n",
      " [ 9.78737984e-01]\n",
      " [ 2.24089320e+00]\n",
      " [ 1.86755799e+00]\n",
      " [ 1.59000000e+02]\n",
      " [ 9.50088418e-01]\n",
      " [-1.51357208e-01]\n",
      " [-1.03218852e-01]\n",
      " [ 1.59000000e+02]\n",
      " [ 1.44043571e-01]\n",
      " [ 1.45427351e+00]\n",
      " [ 7.61037725e-01]]\n"
     ]
    }
   ],
   "source": [
    "# axis = 0 表示把每一行做相加然后再除以总的行数\n",
    "gradient_w = np.mean(gradient_w, axis=0)\n",
    "print('gradient_w ', gradient_w.shape)\n",
    "print('w ', net.w.shape)\n",
    "print(gradient_w)\n",
    "print(net.w)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "我们使用numpy的矩阵操作方便的完成了gradient的计算，但引入了一个问题，gradient_w的形状是(13,)，而w的维度是(13, 1)。导致该问题的原因是使用np.mean函数的时候消除了第0维。为了加减乘除等计算方便，gradient_w和w必须保持一致的形状。所以，我们将gradient_w的维度也设置为(13, 1)，代码如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "gradient_w shape (13, 1)\n"
     ]
    }
   ],
   "source": [
    "gradient_w = gradient_w[:, np.newaxis]\n",
    "print('gradient_w shape', gradient_w.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "综合上面的讨论，我们可以把计算梯度的代码整理如下："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[ 1.59697064],\n",
       "       [-0.92928123],\n",
       "       [ 4.72726926],\n",
       "       [ 1.65712204],\n",
       "       [ 4.96176389],\n",
       "       [ 1.18068454],\n",
       "       [ 4.55846519],\n",
       "       [-3.37770889],\n",
       "       [ 9.57465893],\n",
       "       [10.29870662],\n",
       "       [ 1.3900257 ],\n",
       "       [-0.30152215],\n",
       "       [ 1.09276043]])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "z = net.forward(x)\n",
    "gradient_w = (z - y) * x\n",
    "gradient_w = np.mean(gradient_w, axis=0)\n",
    "gradient_w = gradient_w[:, np.newaxis]\n",
    "gradient_w"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上述代码非常简洁的完成了$w$的梯度计算。同样，计算$b$的梯度的代码也是类似的原理。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-1.0918438870293816e-13"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "gradient_b = (z - y)\n",
    "gradient_b = np.mean(gradient_b)\n",
    "# 此处b是一个数值，所以可以直接用np.mean得到一个标量\n",
    "gradient_b"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "将上面计算$w$和$b$的梯度的过程，写成Network类的gradient函数，代码如下所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        gradient_w = (z-y)*x\n",
    "        gradient_w = np.mean(gradient_w, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = (z - y)\n",
    "        gradient_b = np.mean(gradient_b)\n",
    "        \n",
    "        return gradient_w, gradient_b"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "point [-100.0, -100.0], loss 686.3005008179159\n",
      "gradient [-0.850073323995813, -6.138412364807849]\n"
     ]
    }
   ],
   "source": [
    "# 调用上面定义的gradient函数，计算梯度\n",
    "# 初始化网络，\n",
    "net = Network(13)\n",
    "# 设置[w5, w9] = [-100., +100.]\n",
    "net.w[5] = -100.0\n",
    "net.w[9] = -100.0\n",
    "\n",
    "z = net.forward(x)\n",
    "loss = net.loss(z, y)\n",
    "gradient_w, gradient_b = net.gradient(x, y)\n",
    "gradient_w5 = gradient_w[5][0]\n",
    "gradient_w9 = gradient_w[9][0]\n",
    "print('point {}, loss {}'.format([net.w[5][0], net.w[9][0]], loss))\n",
    "print('gradient {}'.format([gradient_w5, gradient_w9]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 寻找损失函数更小的点\n",
    "\n",
    "下面我们开始研究怎样更新梯度，首先沿着梯度的反方向移动一小步下下一个点P1，观察损失函数的变化。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "point [-99.91499266760042, -99.38615876351922], loss 678.6472185028845\n",
      "gradient [-0.8556356178645292, -6.0932268634065805]\n"
     ]
    }
   ],
   "source": [
    "# 在[w5, w9]平面上，沿着梯度的反方向移动到下一个点P1\n",
    "# 定义移动步长 eta\n",
    "eta = 0.1\n",
    "# 更新参数w5和w9\n",
    "net.w[5] = net.w[5] - eta * gradient_w5\n",
    "net.w[9] = net.w[9] - eta * gradient_w9\n",
    "# 重新计算z和loss\n",
    "z = net.forward(x)\n",
    "loss = net.loss(z, y)\n",
    "gradient_w, gradient_b = net.gradient(x, y)\n",
    "gradient_w5 = gradient_w[5][0]\n",
    "gradient_w9 = gradient_w[9][0]\n",
    "print('point {}, loss {}'.format([net.w[5][0], net.w[9][0]], loss))\n",
    "print('gradient {}'.format([gradient_w5, gradient_w9]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "运行上面的代码，可以发现沿着梯度反方向走一小步，下一个点的损失函数的确减少了。\n",
    "\n",
    "- 读者可以不停的点击上面的代码块，观察损失函数是否一直在变小。\n",
    "\n",
    "将上面的循环的计算过程封装在train和update函数中，如下代码所示。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter 0, point [-99.99144364382136, -99.93861587635192], loss 686.3005008179159\n",
      "iter 50, point [-99.56362583488914, -96.92631128470325], loss 649.221346830939\n",
      "iter 100, point [-99.13580802595692, -94.02279509580971], loss 614.6970095624063\n",
      "iter 150, point [-98.7079902170247, -91.22404911807594], loss 582.543755023494\n",
      "iter 200, point [-98.28017240809248, -88.52620357520894], loss 552.5911329872217\n",
      "iter 250, point [-97.85235459916026, -85.9255316243737], loss 524.6810152322887\n",
      "iter 300, point [-97.42453679022805, -83.41844407682491], loss 498.6667034691001\n",
      "iter 350, point [-96.99671898129583, -81.00148431353688], loss 474.4121018974464\n",
      "iter 400, point [-96.56890117236361, -78.67132338862874], loss 451.7909497114133\n",
      "iter 450, point [-96.14108336343139, -76.42475531364933], loss 430.68610920670284\n",
      "iter 500, point [-95.71326555449917, -74.25869251604028], loss 410.988905460488\n",
      "iter 550, point [-95.28544774556696, -72.17016146534513], loss 392.5985138460824\n",
      "iter 600, point [-94.85762993663474, -70.15629846096763], loss 375.4213919156372\n",
      "iter 650, point [-94.42981212770252, -68.21434557551346], loss 359.3707524354014\n",
      "iter 700, point [-94.0019943187703, -66.34164674796719], loss 344.36607459115214\n",
      "iter 750, point [-93.57417650983808, -64.53564402117185], loss 330.33265059761464\n",
      "iter 800, point [-93.14635870090586, -62.793873918279786], loss 317.2011651461846\n",
      "iter 850, point [-92.71854089197365, -61.11396395304264], loss 304.907305311265\n",
      "iter 900, point [-92.29072308304143, -59.49362926899678], loss 293.3913987080144\n",
      "iter 950, point [-91.86290527410921, -57.930669402782904], loss 282.5980778542974\n",
      "iter 1000, point [-91.43508746517699, -56.4229651670156], loss 272.47596883802515\n",
      "iter 1050, point [-91.00726965624477, -54.968475648286564], loss 262.9774025287022\n",
      "iter 1100, point [-90.57945184731255, -53.56523531604897], loss 254.05814669965383\n",
      "iter 1150, point [-90.15163403838034, -52.21135123828792], loss 245.67715754581488\n",
      "iter 1200, point [-89.72381622944812, -50.90500040003218], loss 237.796349191773\n",
      "iter 1250, point [-89.2959984205159, -49.6444271209092], loss 230.3803798866218\n",
      "iter 1300, point [-88.86818061158368, -48.42794056808474], loss 223.3964536766492\n",
      "iter 1350, point [-88.44036280265146, -47.2539123610643], loss 216.81413643451378\n",
      "iter 1400, point [-88.01254499371925, -46.12077426496303], loss 210.60518520483126\n",
      "iter 1450, point [-87.58472718478703, -45.027015968976976], loss 204.74338990147896\n",
      "iter 1500, point [-87.15690937585481, -43.9711829469081], loss 199.20442646183588\n",
      "iter 1550, point [-86.72909156692259, -42.95187439671279], loss 193.96572062803054\n",
      "iter 1600, point [-86.30127375799037, -41.96774125615467], loss 189.00632158541163\n",
      "iter 1650, point [-85.87345594905815, -41.017484291751295], loss 184.3067847442463\n",
      "iter 1700, point [-85.44563814012594, -40.0998522583068], loss 179.84906300239203\n",
      "iter 1750, point [-85.01782033119372, -39.21364012642417], loss 175.61640587468244\n",
      "iter 1800, point [-84.5900025222615, -38.35768737548557], loss 171.59326591927967\n",
      "iter 1850, point [-84.16218471332928, -37.530876349682856], loss 167.76521193253296\n",
      "iter 1900, point [-83.73436690439706, -36.73213067476985], loss 164.11884842217898\n",
      "iter 1950, point [-83.30654909546485, -35.96041373329276], loss 160.64174090423475\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXoAAAD8CAYAAAB5Pm/hAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl4VeW99vHvLyOEhAwkBAiBMCMyE5HBWURQK7xWrUMLTi9OtVZbq23fc05re86rtdU6FQ89WlFbpU4VrRUVRSvIEOYxEMYQyEASwhxI8pw/9oJGBZLATtbOzv25rlx7rWevlf3L2jt3njxrMuccIiISviL8LkBERBqXgl5EJMwp6EVEwpyCXkQkzCnoRUTCnIJeRCTMKehFRMKcgl5EJMwp6EVEwlyU3wUApKamuqysLL/LEBFpVhYvXrzLOZdW13J1Br2Z9QFm1GrqDvw78JLXngVsAa51zpWbmQFPApcBB4CbnHNLTvYaWVlZ5OTk1FWKiIjUYmZb67NcnUM3zrlc59xg59xgYBiB8H4beAiY7ZzrBcz25gHGA728rynA1IaXLyIiwdLQMfqLgY3Oua3ABGC61z4dmOhNTwBecgHzgSQz6xiUakVEpMEaGvTXAa960+nOuZ3edCGQ7k1nAPm11tnutYmIiA/qHfRmFgNcCbz+9edc4FrHDbresZlNMbMcM8spKSlpyKoiItIADenRjweWOOeKvPmio0My3mOx114AZNZar7PX9hXOuWnOuWznXHZaWp07jUVE5BQ1JOiv51/DNgAzgcne9GTgnVrtkyxgBFBRa4hHRESaWL2OozezNsAlwO21mh8B/mpmtwJbgWu99vcJHFqZR+AInZuDVq2IiDRYvYLeObcfaPe1tlICR+F8fVkH3B2U6uqwekcF763YyU8u7UPg8H0REfm6Zn0JhJwt5Uyds5E567UzV0TkRJp10F8/vAtd28Xx6D/WUV2jm5yLiBxPsw76mKgIHri0D+sK9/L20m8c2CMiIjTzoAe4fEBHBnVO5PEPczl0pNrvckREQk6zD3oz46HxZ7Cj4hDT523xuxwRkZDT7IMeYGSPdlzYJ41nP81j94HDfpcjIhJSwiLoAR4c35e9lVX8Yc5Gv0sREQkpYRP0fTu05dtDO/PivC0U7D7odzkiIiEjbIIe4P5LemPA7z7M9bsUEZGQEVZB3ympNTeNzuLtpQWs3lHhdzkiIiEhrIIe4K4LepIcF8PD764hcDUGEZGWLeyCPrF1NPdd0psFm8uYtbqo7hVERMJc2AU9wPVnZdI7PZ7/en8tlVU6iUpEWrawDPqoyAj+7Yp+bCs7wItzt/hdjoiIr8Iy6AHO7ZXGxX3b8/QneZTsrfS7HBER34Rt0AP87PIzOHSkmsc/Wu93KSIivgnroO+RFs+kkVnMWLSNNTv2+F2OiIgvwjroAe69uBdtW0fz67/rcEsRaZnCPugT46K5/5LezNtYyodrdLiliLQ8YR/0ADcM70Kf9AQefncNBw/rcEsRaVlaRNBHRUbwywlnUrD7IFPn5PldjohIk2oRQQ8wons7JgzuxHOfb2Jr6X6/yxERaTItJugBfnbZGURHGL+YuVo7ZkWkxWhRQZ/ethX3XdKbT3NL+Hhtsd/liIg0iRYV9ACTR2XROz2eX767WjcTF5EWocUFfXRkBL+8sj/byw8yVbcdFJEWoMUFPQRuJn7loE5M/WyjdsyKSNhrkUEP8PPLtWNWRFqGFhv0tXfMvr+y0O9yREQaTb2C3sySzOwNM1tnZmvNbKSZpZjZR2a2wXtM9pY1M3vKzPLMbIWZDW3cH+HU3TQqi/4ZbfnFu6upOHjE73JERBpFfXv0TwIfOOf6AoOAtcBDwGznXC9gtjcPMB7o5X1NAaYGteIgioqM4JGrBlK6r5JHP1jndzkiIo2izqA3s0TgPOB5AOfcYefcbmACMN1bbDow0ZueALzkAuYDSWbWMeiVB0n/jERuGd2NvyzYRs6WMr/LEREJuvr06LsBJcCfzGypmf2PmbUB0p1zO71lCoF0bzoDyK+1/nav7SvMbIqZ5ZhZTklJyan/BEFw3yW9yUhqzU/fWsnhqhpfaxERCbb6BH0UMBSY6pwbAuznX8M0ALjAYSsNOnTFOTfNOZftnMtOS0tryKpB1yY2il9NPJMNxfuY9rmOrReR8FKfoN8ObHfOLfDm3yAQ/EVHh2S8x6PXFCgAMmut39lrC2kX9U3n8gEdeeqTPDbv0rH1IhI+6gx651whkG9mfbymi4E1wExgstc2GXjHm54JTPKOvhkBVNQa4glp//GtfsRGRfDzt1fq2HoRCRv1PermHuDPZrYCGAz8F/AIcImZbQDGePMA7wObgDzgj8BdQa24EbVv24qHxvdl3sZSZizKr3sFEZFmIKo+CznnlgHZx3nq4uMs64C7T7Mu31x/VhfeW76T//z7Ws7rnUanpNZ+lyQiclpa7JmxJxIRYTz67YFU1Th++paGcESk+VPQH0eXdnE8OK4Pn60v4Y3F2/0uR0TktCjoT2DSyCyGZ6Xw8HtrKKw45Hc5IiKnTEF/AhERxm+uHsiR6hodhSMizZqC/iSyUtvw47F9mL2umL8tC/lTAUREjktBX4ebR3djaJckfjFzDcV7NIQjIs2Pgr4OkRHGY9cM4tCRah58c4WGcESk2VHQ10OPtHgeGt+XT3NL+MvCbX6XIyLSIAr6epo8Motze6Xy6/fW6lo4ItKsKOjrKSLCeOzqQcRERXDfjGVUVetyxiLSPCjoG6BDYit+PbE/y/J384c5upyxiDQPCvoG+tagTkwY3IknZ29gef5uv8sREamTgv4UPHxlf9onxHLfjGUcPFztdzkiIieloD8FiXHR/O6aQWzatZ//en+t3+WIiJyUgv4UjeqZym3ndOPl+Vv5cHWh3+WIiJyQgv40PDCuD2d2astP3lzBzoqDfpcjInJcCvrTEBsVyTM3DOVIVQ33vraM6hqdNSsioUdBf5q6pbbhVxP7s3BzGU9/ssHvckREvkFBHwRXDe3MVUMyeGr2BuZvKvW7HBGRr1DQB8nDE/vTtV0bfvjaMsr3H/a7HBGRYxT0QRIfG8XT1w+hdH8lD7yhq1yKSOhQ0AdR/4xEHhp/Bh+vLeKFuVv8LkdEBFDQB90to7MY2y+d///+WnK2lPldjoiIgj7YzAI3KslIbs3df1nCrn2VfpckIi2cgr4RJLaOZuqNw9h94Ag/eHWpjq8XEV8p6BtJv05t+dXE/szbWMrjH+X6XY6ItGAK+kZ0bXYm38nO5NlPNzJ7bZHf5YhIC6Wgb2S/nHAm/Tq25b4Zy8gvO+B3OSLSAtUr6M1si5mtNLNlZpbjtaWY2UdmtsF7TPbazcyeMrM8M1thZkMb8wcIda2iI3nuu8NwwB2vLNb160WkyTWkR3+hc26wcy7bm38ImO2c6wXM9uYBxgO9vK8pwNRgFdtcdWkXx5PXDWbNzj08+KZOphKRpnU6QzcTgOne9HRgYq32l1zAfCDJzDqexuuEhYv6pvPjsX2YuXwH0z7f5Hc5ItKC1DfoHfChmS02syleW7pzbqc3XQike9MZQH6tdbd7bS3eXRf04PIBHXnkg3XMyS32uxwRaSHqG/TnOOeGEhiWudvMzqv9pAuMRTRoPMLMpphZjpnllJSUNGTVZitwMtVA+nZoyz2vLmVTyT6/SxKRFqBeQe+cK/Aei4G3geFA0dEhGe/xaBe1AMistXpnr+3r33Oacy7bOZedlpZ26j9BMxMXE8W07w0jKsKY8vJi9h464ndJIhLm6gx6M2tjZglHp4GxwCpgJjDZW2wy8I43PROY5B19MwKoqDXEI0BmShzP3jiUzbv2c9+MZdTozFkRaUT16dGnA1+Y2XJgIfB359wHwCPAJWa2ARjjzQO8D2wC8oA/AncFveowMKpHKv92+Rl8vLaY336oM2dFpPFE1bWAc24TMOg47aXAxcdpd8DdQakuzE0elUVu0T7+MGcjWaltuDY7s+6VREQaqM6gl8ZjZjw84Uzyyw7ws7dW0jm5NaN6pPpdloiEGV0CwWfRkRE8e+NQuqW24c5XlrBRR+KISJAp6ENAYutoXrjpLKIijFteXESZ7jkrIkGkoA8RmSlxTJuUzc6KQ9z+cg6VVbomjogEh4I+hAzrmszj1w5i0ZZyHtQNxkUkSLQzNsRcMbATW0sP8NisXDomtebBcX39LklEmjkFfQi664IeFOw+yNQ5G2mfEMvNo7v5XZKINGMK+hBkZvxqQn927a3k4ffWkJYQyxUDO/ldlog0UxqjD1GREcZT1w8hu2sy989YzryNu/wuSUSaKQV9CGsVHcn/TDqLrNQ4bn9pMWt27PG7JBFphhT0IS4xLpoXbx5OfKsobvrTQt13VkQaTEHfDHRKas30W4Zz6Eg1k15YSMneSr9LEpFmREHfTPROT+CFm86isOIQ33t+AbsP6OxZEakfBX0zkp2VwrRJw9hUsp/Jf1rEvsoqv0sSkWZAQd/MnNsrjWduGMKqggpum76IQ0d0qQQROTkFfTM09swOPH7tIBZsLuPOVxZzuKrG75JEJIQp6JupCYMz+M+JA/g0t4T7ZiyjqlphLyLHpzNjm7Ebzu7CgcNV/Prva4mNjuCxqwcRGWF+lyUiIUZB38zddm53Dhyu5vGP1mMYv7l6oMJeRL5CQR8GfnBxL5yDJz5eD6CwF5GvUNCHiXvH9AIU9iLyTQr6MKKwF5HjUdCHGYW9iHydgj4M1Q77Gud47OqBREXqSFqRlkpBH6buHdOLqEjjsVm5HDhcxVPXDyE2KtLvskTEB+rmhbG7L+zJf3yrH7NWF3Hb9BwOHtblEkRaIgV9mLt5dDd+8+2BzM3bxaQXFrDn0BG/SxKRJqagbwGuPSuTp64fwtJtu7nxjwso369LHIu0JAr6FuKKgZ2YNmkYuUV7+c60Lynec8jvkkSkidQ76M0s0syWmtl73nw3M1tgZnlmNsPMYrz2WG8+z3s+q3FKl4a6qG86L958FgXlB7lq6jw2lezzuyQRaQIN6dHfC6ytNf8o8IRzridQDtzqtd8KlHvtT3jLSYgY1SOVV6eM4ODhaq5+7kuWbiv3uyQRaWT1Cnoz6wxcDvyPN2/ARcAb3iLTgYne9ARvHu/5i73lJUQM7JzEm3eOIj42ihv+uIBP1xX7XZKINKL69uh/D/wEOHrR83bAbufc0XvZbQcyvOkMIB/Ae77CW/4rzGyKmeWYWU5JSckpli+nKiu1DW/eOYoe7dtw20s5vJ6T73dJItJI6gx6M7sCKHbOLQ7mCzvnpjnnsp1z2WlpacH81lJPaQmxvDZlJKN6tOOBN1bw7Kd5OOf8LktEgqw+PfrRwJVmtgV4jcCQzZNAkpkdPbO2M1DgTRcAmQDe84lAaRBrliCKj43i+clnMXFwJx6blcu/vbNKd6sSCTN1Br1z7qfOuc7OuSzgOuAT59yNwKfA1d5ik4F3vOmZ3jze8584dRNDWkxUBI9fO5g7zu/BK/O3ccv0HJ1YJRJGTuc4+geB+80sj8AY/PNe+/NAO6/9fuCh0ytRmkJEhPHQ+L48+u0BzMvbxdVT55FfdsDvskQkCCwUOtvZ2dkuJyfH7zLEMy9vF3e8spiYqAimTcpmaJdkv0sSkeMws8XOuey6ltOZsfINo3qm8tZdo2kTG8V10+bz7vIdfpckIqdBQS/H1bN9PG/fNZrBnZO459WlPPnxBmpq/P/vT0QaTkEvJ5TSJoaXbxvOVUMzeOLj9dz15yXsq6yqe0URCSkKejmp2KhIfnfNIP7f5Wfw4ZpCrvrDXLbs2u93WSLSAAp6qZOZcdu53XnplrMp3lvJlc98wZxcXTZBpLlQ0Eu9ndMrlXe/fw4ZyXHc/OIi/jBHZ9KKNAcKemmQzJQ43rxzJJcP6MhvPsjl+68uZb/G7UVCmoJeGiwuJoqnrx/CT8f35R8rd3LlM1+wvmiv32WJyAko6OWUmBm3n9+DV249m4qDVUx4Zi5vLt7ud1kichwKejkto3qm8v4PzmFg50R+9PpyHnxjBYeOVPtdlojUoqCX09a+bSv+fNvZ3H1hD2bk5DPx2bm6TaFICFHQS1BERUbwwKV9+dPNZ1G45xBXPjNXl04QCREKegmqC/u05/0fnEvv9HjueXUpP359uc6mFfGZgl6CrlNSa2bcPpIfXNSTt5Zs5/Kn/smy/N1+lyXSYinopVFER0Zw/9g+vDZlJFXVjqunzuPZT/Oo1oXRRJqcgl4a1fBuKbx/77lc2r8Dj83K5YY/zmfH7oN+lyXSoijopdElto7mmeuH8NjVA1lZUMH4J//JO8sKdPkEkSaioJcmYWZck53J+z84l+5pbbj3tWXc9ecllO6r9Ls0kbCnoJcmlZXahjfuGMWD4/oye20xY5/4nA9W7fS7LJGwpqCXJhcZYdx5QQ/eveccOia14o5XlnDva0vZfeCw36WJhCUFvfimT4cE3r5rNPeN6c3fV+xk7BOfM3ttkd9liYQdBb34KjoygnvH9OJvd48mOS6GW6fncM+rS9mlsXuRoFHQS0jon5HIzHsCvftZqwoZ8/hnvJ6TryNzRIJAQS8hIzYqknvH9OL9e8+hZ1o8D7yxgu8+v4CtpbpHrcjpUNBLyOnZPoG/3j6SX0/sz4r8CsY+8TnPfbaRquoav0sTaZYU9BKSIiKM747oykf3n8/5vdN45B/ruOLpL1i0pczv0kSaHQW9hLQOia2YNimb5747jD0Hj3DNc1/yo78up2SvdtaK1JeCXpqFcf078PGPzueuC3owc3kBF/1uDi/O3azhHJF6qDPozayVmS00s+VmttrMfum1dzOzBWaWZ2YzzCzGa4/15vO857Ma90eQliIuJoqfjOvLBz88j8GZSfzi3TVc8fQX5Gg4R+Sk6tOjrwQucs4NAgYD48xsBPAo8IRzridQDtzqLX8rUO61P+EtJxI0PdLieemW4Uy9cSh7Dh7h6ue+5P4Zy9hZoatiihxPnUHvAo7eADTa+3LARcAbXvt0YKI3PcGbx3v+YjOzoFUsQuAiaeMHdDw2nPPeyp1c+Ns5PPHReg4c1h2tRGqr1xi9mUWa2TKgGPgI2Ajsds4d/Y3aDmR40xlAPoD3fAXQLphFixx1dDhn9v3nc/EZ6Tw5ewMX/nYObyzeTo1uciIC1DPonXPVzrnBQGdgOND3dF/YzKaYWY6Z5ZSUlJzut5MWLjMljmdvGMqbd46kQ2Jrfvz6cq589gsWbCr1uzQR3zXoqBvn3G7gU2AkkGRmUd5TnYECb7oAyATwnk8EvvHb5pyb5pzLds5lp6WlnWL5Il81rGsKb985it9/ZzCl+w7znWnzuf3lHPKK99W9skiYqs9RN2lmluRNtwYuAdYSCPyrvcUmA+940zO9ebznP3G6YIk0oYgIY+KQDD750QX86JLefLFhF2Of+IwH31ih2xhKi2R1ZbCZDSSwczWSwB+GvzrnHjaz7sBrQAqwFPiuc67SzFoBLwNDgDLgOufcppO9RnZ2tsvJyTntH0bkeEr3VfLspxt5Zf5WMJg0oit3XdiTlDYxfpcmclrMbLFzLrvO5UKhs62gl6awvfwAv/94A28t2U5cTBRTzuvOred0o01sVN0ri4QgBb3ICWwo2stvP8xl1uoi2rWJ4c4LenDj2V1pHRPpd2kiDaKgF6nDkm3l/HZWLvM2lpIaH8vt53XnxhFdiItRD1+aBwW9SD0t3FzGk7PXMzevlNT4GKac153vjuiqwJeQp6AXaaCcLWU8OXsD/9ywi3ZtYvi/53XneyO6agxfQpaCXuQULd5axu8/DgR+SpsYbh6VxfdGdiUpTkfpSGhR0IucpiXbynl69gY+zS0hLiaS64d34bZzu9ExsbXfpYkACnqRoFlXuIf//mwTM5fvIMJgwuAM7ji/Oz3bJ/hdmrRwCnqRIMsvO8DzX2zmtUXbOHSkhjFnpHPnBd0Z1jXF79KkhVLQizSSsv2HmT5vC9O/3MLuA0cYnJnELed0Y3z/DkRH6qZt0nQU9CKN7MDhKl7P2c6L87awedd+OrRtxfdGduWG4V1I1uUVpAko6EWaSE2NY876Yl74Ygtf5O0iNiqCq4ZmcPPobvRO1zi+NJ76Br0OEBY5TRERxkV907mobzq5hXt5cd5m3lpSwKsL8zmnZyqTRnblor7tidKwjvhEPXqRRlC2/zCvLtzGy19upXDPITq0bcV1wzO57qwudEhs5Xd5EiY0dCMSAqqqa/hkXTGvLNjG5+tLiIwwxpzRnhvP7so5PVOJiNDtlOXUaehGJARERUYw9swOjD2zA9tKD/CXhdt4PSefWauL6JISxw1nd+GaYZ1pFx/rd6kSxtSjF2lilVXVzFpdxCvzt7JwcxnRkcbFfdO5Jrsz5/dO01i+1Jt69CIhKjYqkisHdeLKQZ3YULSXGYvyeXtpAR+sLiQtIZarhmZwzbBMeraP97tUCRPq0YuEgCPeWP7rOdv5NLeY6hrHkC5JXDMskysGdaRtq2i/S5QQpJ2xIs1Uyd5K/ra0gL/m5LOheB+toiMY268DE4d04txeaTr7Vo5R0Is0c845lm+v4PWcfN5bsZOKg0dIjovm8oEdmTg4g6FdknXUTgunoBcJI4eravh8fQl/W1bAx2uLOHSkhoyk1kwY3IkJgzPo00Fn4LZECnqRMLWvsooPVxfyzrIdfJG3i+oaR98OCVw5uBOX9e9IVmobv0uUJqKgF2kBdu2r5O8rdvK3ZQUs3bYbgH4d23LZgA6MH9CRHmk6ciecKehFWpjt5Qf4YFUh/1hVyOKt5QD0SU9g/IAOXDagoy6wFoYU9CItWGHFIT5YtZP3VxWyaEsZzkHP9vFc1j9wlu6Zndpiph25zZ2CXkQAKN57iFmri/jHyp3M31RKjYOOia0Yc0Y6Y/qlM6J7CrFRkX6XKadAQS8i31C6r5JP1hXz8doiPl+/i4NHqmkTE8n5fdIYc0Y6F/Zpr5umNCMKehE5qUNHqvlyYykfrS1i9toiivZUEmGQnZXCJWekc2HfNHqkxWuIJ4QFLejNLBN4CUgHHDDNOfekmaUAM4AsYAtwrXOu3AKfiieBy4ADwE3OuSUnew0FvYi/amocq3ZU8PGaIj5aW8zanXsAyEhqzfl90ji/dxqjerQjQZdiCCnBDPqOQEfn3BIzSwAWAxOBm4Ay59wjZvYQkOyce9DMLgPuIRD0ZwNPOufOPtlrKOhFQkvB7oN8llvCZ+uLmZtXyr7KKqIijGFdk7mgT3vO753GGR0T1Nv3WaMN3ZjZO8Az3tcFzrmd3h+DOc65Pmb23970q97yuUeXO9H3VNCLhK4j1TUs3lrOZ+tL+Cy3hDVeb799Qizn9U7j3F6pjOqRSlqCrqnf1BrlMsVmlgUMARYA6bXCu5DA0A5ABpBfa7XtXtsJg15EQld0ZAQjurdjRPd2PDiuL8V7DgVCf30JH60p4o3F24HAMfujerZjdI9UhndP0RU3Q0i9g97M4oE3gR865/bU/pfNOefMrEH/GpjZFGAKQJcuXRqyqoj4qH3bVlyTnck12ZlU1zhWFVQwd+Mu5uWV8pcF2/jT3C1ERhgDMhIZ7QX/0K7JtIrWIZx+qdfQjZlFA+8Bs5xzj3ttx4ZkNHQjIhA4kmfptt3M27iLuXm7WL69guoaR0xUBNldkxnVox1nZaUwKDNJwR8EwdwZa8B0Ajtef1ir/TGgtNbO2BTn3E/M7HLg+/xrZ+xTzrnhJ3sNBb1IeNp76AgLN5cxN6+UeRt3sa5wLwAxkREMzkzirG7JDO/WjmFdk4mP1Q3vGiqYQX8O8E9gJVDjNf+MwDj9X4EuwFYCh1eWeX8YngHGETi88mbn3ElTXEEv0jKU7z9MztZyFm4uZeGWclYVBHr8EQZndkpkeLcUzspKYXi3FFJ04laddMKUiIS8/ZVVLNlWzqLNZSzYXMay/N1UVgX6kz3bxzO0SxJDuyQztGsyPdPidaOVr1HQi0izU1lVzcrtFSzcUsaizWUszd/N7gNHAEhoFcXgzCSGdElmaJckhmQmkxjXso/saZTDK0VEGlNsVCTZWSlkZ6XABYHbKW7etZ8l23azZFs5S7aW88wnG6jx+qc928czJDOJoV2TGdolmZ7t44lUr/8b1KMXkWZlX2UVK/K94N+2m6Xbyin3ev2toyPpn9GWARlJDOycyMDOiWS1axO2Qz4auhGRFuFor3/ptt2sLKhgZUEFq3dUcOhIYKw/ITaK/hmB0B/QOZFBnZPonNw6LC7foKEbEWkRzIzuafF0T4vn28M6A1BVXcOG4n2s3F7BioLdrNxewZ/mbuFwdSD8k+KiGZCRyICMRM7slMgZHRPCu+evHr2ItASVVdWsL9x3LPhXbK9gfdFeqrwB/7iYSPp2SKBfp7b065hIv05t6ZOeQOuY0D2xS0M3IiJ1qKyqZkPRPtbs3MOaHXtYs3MPa3fsYW9lFQARBt1S29CvUyL9Orb1/gi0DZkLuGnoRkSkDrFRkfTPSKR/RuKxNucc28sPsvpo8O/cw5Kt5by7fMexZdq1iaF3egJ9OiR4j/H0Sk8I2Qu5KehFRGoxMzJT4shMiWNc/w7H2isOHAn0/HfuYX3hXnKL9vJ6Tj77D1cfW6ZTYit61f4DkJ5Az/bxvg//KOhFROohMS6akT3aMbJHu2NtNTWOgt0H2VC8l9zCfawv2ktu4V6+3FTKYe8MXzPomhJH7/RA+PdsH0/P9vH0SGu6PwAKehGRUxQR8a/e/0V904+1V1XXsK3sgBf83h+Aor3MXldMtbfz1yxwq8YHLu3DhMEZjVqngl5EJMiiIiOOHfI5rv+/2g9X1bC1dD8biveRV7yPDcX7SItv/B27CnoRkSYSExVBr/QEeqUnNOnrRjTpq4mISJNT0IuIhDkFvYhImFPQi4iEOQW9iEiYU9CLiIQ5Bb2ISJhT0IuIhLmQuEyxmZUAW09x9VRgVxDLCRbV1TChWheEbm2qq2HCsa6uzrm0uhYKiaA/HWaWU5/rMTc11dUwoVoXhG5tqqthWnJdGroREQlzCnoRkTAXDkE/ze8CTkB1NUyo1gWhW5vqapiF9IW5AAAE+ElEQVQWW1ezH6MXEZGTC4cevYiInESzDnozG2dmuWaWZ2YPNfFrZ5rZp2a2xsxWm9m9XvsvzKzAzJZ5X5fVWuenXq25ZnZpI9a2xcxWeq+f47WlmNlHZrbBe0z22s3MnvLqWmFmQxuppj61tskyM9tjZj/0Y3uZ2QtmVmxmq2q1NXj7mNlkb/kNZja5kep6zMzWea/9tpklee1ZZnaw1nZ7rtY6w7z3P8+r3Rqhrga/b8H+fT1BXTNq1bTFzJZ57U25vU6UDf59xpxzzfILiAQ2At2BGGA50K8JX78jMNSbTgDWA/2AXwA/Ps7y/bwaY4FuXu2RjVTbFiD1a22/AR7yph8CHvWmLwP+ARgwAljQRO9dIdDVj+0FnAcMBVad6vYBUoBN3mOyN53cCHWNBaK86Udr1ZVVe7mvfZ+FXq3m1T6+Eepq0PvWGL+vx6vra8//Dvh3H7bXibLBt89Yc+7RDwfynHObnHOHgdeACU314s65nc65Jd70XmAtcLIbP04AXnPOVTrnNgN5BH6GpjIBmO5NTwcm1mp/yQXMB5LMrGMj13IxsNE5d7KT5BpteznnPgfKjvN6Ddk+lwIfOefKnHPlwEfAuGDX5Zz70DlX5c3OBzqf7Ht4tbV1zs13gbR4qdbPErS6TuJE71vQf19PVpfXK78WePVk36ORtteJssG3z1hzDvoMIL/W/HZOHrSNxsyygCHAAq/p+96/YC8c/feMpq3XAR+a2WIzm+K1pTvndnrThcDROxn7sR2v46u/gH5vL2j49vFju91CoOd3VDczW2pmn5nZuV5bhldLU9TVkPetqbfXuUCRc25DrbYm315fywbfPmPNOehDgpnFA28CP3TO7QGmAj2AwcBOAv8+NrVznHNDgfHA3WZ2Xu0nvZ6LL4dbmVkMcCXwutcUCtvrK/zcPidiZj8HqoA/e007gS7OuSHA/cBfzKxtE5YUcu/b11zPVzsTTb69jpMNxzT1Z6w5B30BkFlrvrPX1mTMLJrAG/ln59xbAM65IudctXOuBvgj/xpuaLJ6nXMF3mMx8LZXQ9HRIRnvsbip6/KMB5Y454q8Gn3fXp6Gbp8mq8/MbgKuAG70AgJvaKTUm15MYPy7t1dD7eGdRqnrFN63ptxeUcBVwIxa9Tbp9jpeNuDjZ6w5B/0ioJeZdfN6idcBM5vqxb0xwOeBtc65x2u11x7f/j/A0SMCZgLXmVmsmXUDehHYCRTsutqYWcLRaQI781Z5r390r/1k4J1adU3y9vyPACpq/XvZGL7S0/J7e9XS0O0zCxhrZsnesMVYry2ozGwc8BPgSufcgVrtaWYW6U13J7B9Nnm17TGzEd5ndFKtnyWYdTX0fWvK39cxwDrn3LEhmabcXifKBvz8jJ3O3mW/vwjsrV5P4K/zz5v4tc8h8K/XCmCZ93UZ8DKw0mufCXSstc7PvVpzOc09+yepqzuBIxqWA6uPbhegHTAb2AB8DKR47QY869W1EshuxG3WBigFEmu1Nfn2IvCHZidwhMC4562nsn0IjJnneV83N1JdeQTGaY9+xp7zlv229/4uA5YA36r1fbIJBO9G4Bm8EyODXFeD37dg/74ery6v/UXgjq8t25Tb60TZ4NtnTGfGioiEueY8dCMiIvWgoBcRCXMKehGRMKegFxEJcwp6EZEwp6AXEQlzCnoRkTCnoBcRCXP/C+Afh4uuEJ1mAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights,1)\n",
    "        self.w[5] = -100.\n",
    "        self.w[9] = -100.\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        gradient_w = (z-y)*x\n",
    "        gradient_w = np.mean(gradient_w, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = (z - y)\n",
    "        gradient_b = np.mean(gradient_b)        \n",
    "        return gradient_w, gradient_b\n",
    "    \n",
    "    def update(self, graident_w5, gradient_w9, eta=0.01):\n",
    "        net.w[5] = net.w[5] - eta * gradient_w5\n",
    "        net.w[9] = net.w[9] - eta * gradient_w9\n",
    "        \n",
    "    def train(self, x, y, iterations=100, eta=0.01):\n",
    "        points = []\n",
    "        losses = []\n",
    "        for i in range(iterations):\n",
    "            points.append([net.w[5][0], net.w[9][0]])\n",
    "            z = self.forward(x)\n",
    "            L = self.loss(z, y)\n",
    "            gradient_w, gradient_b = self.gradient(x, y)\n",
    "            gradient_w5 = gradient_w[5][0]\n",
    "            gradient_w9 = gradient_w[9][0]\n",
    "            self.update(gradient_w5, gradient_w9, eta)\n",
    "            losses.append(L)\n",
    "            if i % 50 == 0:\n",
    "                print('iter {}, point {}, loss {}'.format(i, [net.w[5][0], net.w[9][0]], L))\n",
    "        return points, losses\n",
    "\n",
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "x = train_data[:, :-1]\n",
    "y = train_data[:, -1:]\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "num_iterations=2000\n",
    "# 启动训练\n",
    "points, losses = net.train(x, y, iterations=num_iterations, eta=0.01)\n",
    "\n",
    "# 画出损失函数的变化趋势\n",
    "plot_x = np.arange(num_iterations)\n",
    "plot_y = np.array(losses)\n",
    "plt.plot(plot_x, plot_y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 对所有参数计算梯度并更新\n",
    "\n",
    "为了能给读者直观的感受，上面演示的梯度下降法的过程仅包含$w_5$和$w_9$两个参数。房价预测的完整模型，必须要对所有参数$w$和$b$进行求解。这需要将Network中的update和train函数进行修改。由于不在限定参与计算的参数（所有参数均参与计算），修改之后的代码反而更加简洁。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "iter 9, loss 1.8984947314576224\n",
      "iter 19, loss 1.8031783384598725\n",
      "iter 29, loss 1.7135517565541092\n",
      "iter 39, loss 1.6292649416831264\n",
      "iter 49, loss 1.5499895293373231\n",
      "iter 59, loss 1.4754174896452612\n",
      "iter 69, loss 1.4052598659324693\n",
      "iter 79, loss 1.3392455915676864\n",
      "iter 89, loss 1.2771203802372915\n",
      "iter 99, loss 1.218645685090292\n",
      "iter 109, loss 1.1635977224791534\n",
      "iter 119, loss 1.111766556287068\n",
      "iter 129, loss 1.0629552390811503\n",
      "iter 139, loss 1.0169790065644477\n",
      "iter 149, loss 0.9736645220185994\n",
      "iter 159, loss 0.9328491676343147\n",
      "iter 169, loss 0.8943803798194307\n",
      "iter 179, loss 0.8581150257549611\n",
      "iter 189, loss 0.8239188186389669\n",
      "iter 199, loss 0.7916657692169988\n",
      "iter 209, loss 0.761237671346902\n",
      "iter 219, loss 0.7325236194855752\n",
      "iter 229, loss 0.7054195561163928\n",
      "iter 239, loss 0.6798278472589763\n",
      "iter 249, loss 0.6556568843183528\n",
      "iter 259, loss 0.6328207106387195\n",
      "iter 269, loss 0.6112386712285092\n",
      "iter 279, loss 0.59083508421862\n",
      "iter 289, loss 0.5715389327049418\n",
      "iter 299, loss 0.5532835757100347\n",
      "iter 309, loss 0.5360064770773406\n",
      "iter 319, loss 0.5196489511849665\n",
      "iter 329, loss 0.5041559244351539\n",
      "iter 339, loss 0.48947571154034963\n",
      "iter 349, loss 0.47555980568755696\n",
      "iter 359, loss 0.46236268171965056\n",
      "iter 369, loss 0.44984161152579916\n",
      "iter 379, loss 0.43795649088328303\n",
      "iter 389, loss 0.4266696770400226\n",
      "iter 399, loss 0.41594583637124666\n",
      "iter 409, loss 0.4057518014851036\n",
      "iter 419, loss 0.3960564371908221\n",
      "iter 429, loss 0.38683051477942226\n",
      "iter 439, loss 0.3780465941011246\n",
      "iter 449, loss 0.3696789129556087\n",
      "iter 459, loss 0.3617032833413179\n",
      "iter 469, loss 0.3540969941381648\n",
      "iter 479, loss 0.3468387198244131\n",
      "iter 489, loss 0.3399084348532937\n",
      "iter 499, loss 0.33328733333814486\n",
      "iter 509, loss 0.3269577537166779\n",
      "iter 519, loss 0.32090310808539985\n",
      "iter 529, loss 0.3151078159144129\n",
      "iter 539, loss 0.30955724187078903\n",
      "iter 549, loss 0.3042376374955925\n",
      "iter 559, loss 0.2991360864954391\n",
      "iter 569, loss 0.2942404534243286\n",
      "iter 579, loss 0.2895393355454012\n",
      "iter 589, loss 0.28502201767532415\n",
      "iter 599, loss 0.28067842982626157\n",
      "iter 609, loss 0.27649910747186535\n",
      "iter 619, loss 0.2724751542744919\n",
      "iter 629, loss 0.2685982071209627\n",
      "iter 639, loss 0.26486040332365085\n",
      "iter 649, loss 0.2612543498525749\n",
      "iter 659, loss 0.2577730944725093\n",
      "iter 669, loss 0.2544100986669443\n",
      "iter 679, loss 0.2511592122380609\n",
      "iter 689, loss 0.2480146494787638\n",
      "iter 699, loss 0.24497096681926708\n",
      "iter 709, loss 0.2420230418567802\n",
      "iter 719, loss 0.23916605368251415\n",
      "iter 729, loss 0.23639546442555454\n",
      "iter 739, loss 0.23370700193813704\n",
      "iter 749, loss 0.2310966435515475\n",
      "iter 759, loss 0.2285606008362593\n",
      "iter 769, loss 0.22609530530403904\n",
      "iter 779, loss 0.22369739499361888\n",
      "iter 789, loss 0.2213637018851542\n",
      "iter 799, loss 0.21909124009208833\n",
      "iter 809, loss 0.21687719478222933\n",
      "iter 819, loss 0.21471891178284025\n",
      "iter 829, loss 0.21261388782734392\n",
      "iter 839, loss 0.2105597614038757\n",
      "iter 849, loss 0.20855430416838638\n",
      "iter 859, loss 0.20659541288730932\n",
      "iter 869, loss 0.20468110187697833\n",
      "iter 879, loss 0.2028094959090178\n",
      "iter 889, loss 0.20097882355283644\n",
      "iter 899, loss 0.19918741092814596\n",
      "iter 909, loss 0.19743367584210875\n",
      "iter 919, loss 0.1957161222872899\n",
      "iter 929, loss 0.19403333527807176\n",
      "iter 939, loss 0.19238397600456975\n",
      "iter 949, loss 0.19076677728439412\n",
      "iter 959, loss 0.1891805392938162\n",
      "iter 969, loss 0.18762412556104593\n",
      "iter 979, loss 0.18609645920539716\n",
      "iter 989, loss 0.18459651940712488\n",
      "iter 999, loss 0.18312333809366155\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAX4AAAD8CAYAAABw1c+bAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJzt3Xl8VfWd//HXJ/u+kRC2sAooighEFLWirSJaq611qrZabJkHw3RfZjp1Nqd2fjOddrqOy5RRa+uvo7XVWqpWtC7FDSUoCmHfw5qwJQGyks/8cQ80IJAbuMm5uff9fDzu497zPd+bfE4OvM+553zvOebuiIhI8kgJuwAREeldCn4RkSSj4BcRSTIKfhGRJKPgFxFJMgp+EZEko+AXEUkyCn4RkSSj4BcRSTJpYRdwPKWlpT58+PCwyxAR6TMWL168y93Loukbl8E/fPhwqqqqwi5DRKTPMLNN0fbVoR4RkSSj4BcRSTIKfhGRJNNl8JtZhZm9ZGbLzazazL58nD5mZj8xs7Vm9p6ZTeo0b6aZrQkeM2O9ACIi0j3RnNxtB77u7m+bWT6w2Myed/flnfpcDYwOHhcA9wEXmFkJcCdQCXjw3nnuvjemSyEiIlHrco/f3be7+9vB60ZgBTD4mG7XA7/wiIVAkZkNBK4Cnnf3PUHYPw/MiOkSiIhIt3TrGL+ZDQcmAm8eM2swUNNpekvQdqJ2EREJSdTBb2Z5wOPAV9y9IdaFmNlsM6sys6q6urpuv7+57RA//dM6Xl2zK9aliYgklKiC38zSiYT+L939ieN02QpUdJoeErSdqP193H2uu1e6e2VZWVRfPjtKRmoKcxes59eLa7ruLCKSxKIZ1WPAA8AKd//BCbrNAz4djO65EKh39+3AfGC6mRWbWTEwPWiLuZQU4wOjS3llzS46OnQDeRGRE4lmVM/FwG3AUjNbErT9PTAUwN3/G3gGuAZYCxwEPhPM22Nm3wYWBe+7y933xK78o106pownl2yjelsD44cU9tSvERHp07oMfnd/FbAu+jjw+RPMexB48JSq66YPjI4cIlqwpk7BLyJyAgn1zd2y/EzGDSzgT6u7f3JYRCRZJFTwQ+Rwz9ub9tLY3BZ2KSIicSkBg7+U9g7njXW7wy5FRCQuJVzwTx5WTHZ6Kq9oPL+IyHElXPBnpqUydVQ/FqzRcX4RkeNJuOAHuHR0KZt2H2TT7gNhlyIiEncSM/jHBMM6NbpHROR9EjL4R5TmMqQ4mz+t1nF+EZFjJWTwmxnTxpTx+rpdtLQfCrscEZG4kpDBD/DBM/tzsPUQb67vsStEiIj0SQkb/BeNKiUzLYUXV9aGXYqISFxJ2ODPzkjl4jNKeWHlTiKXEhIREUjg4IfI4Z6aPU2sq9sfdikiInEj4YMf4IUVOtwjInJYQgf/oKJszhpYwAs6zi8ickRCBz/Ah87sz+JNe9l3sDXsUkRE4kLCB/8Hz+rPoQ7XNfpFRAIJH/wThhTRLzdDwzpFRAIJH/ypKcZlY/vz8qo62g91hF2OiEjougx+M3vQzGrNbNkJ5v+tmS0JHsvM7JCZlQTzNprZ0mBeVayLj9YHz+xPfVMbVZv2hlWCiEjciGaP/yFgxolmuvv33P08dz8PuAP4k7t3vk7C5cH8ytMr9dRdOqaUjNQUnqveGVYJIiJxo8vgd/cFQLQXvLkFeOS0KuoB+VnpXDK6lPnVO/QtXhFJejE7xm9mOUQ+GTzeqdmB58xssZnN7uL9s82sysyq6upiPwLnqrPL2bqvieptDTH/2SIifUksT+5+BHjtmMM8l7j7JOBq4PNmdumJ3uzuc9290t0ry8rKYlhWxBVnlZNiML96R8x/tohIXxLL4L+ZYw7zuPvW4LkW+C0wJYa/r1v65WVSObxEwS8iSS8mwW9mhcA04Hed2nLNLP/wa2A6cNyRQb1lxtkDWL1zPxt26V68IpK8ohnO+QjwBjDWzLaY2Swzm2Nmczp1+xjwnLt3TtRy4FUzexd4C3ja3Z+NZfHdNf3sckCHe0QkuaV11cHdb4miz0NEhn12blsPTDjVwnrCkOIczhlcwPzqHcyZNirsckREQpHw39w91lXjBvDO5n3sbGgOuxQRkVAkXfDPOGcAAM/pcI+IJKmkC/4z+ucxsiyXZ5Yq+EUkOSVd8JsZ144fyJsbdlPbqMM9IpJ8ki74Aa6dMIgOhz9or19EklBSBv+Y8nzGlOfx1Hvbwi5FRKTXJWXwA1x77iAWbdzL9vqmsEsREelVSRz8AwF4+r3tIVciItK7kjb4R5blMW5gAU8p+EUkySRt8ANcO2EgS2r2UbPnYNiliIj0muQO/vGDAHh6qfb6RSR5JHXwD+2Xw4QhhRrdIyJJJamDH+AjEwaxbGsDa2v3h12KiEivSPrgv27CIFIMfvvOlrBLERHpFUkf/P0LsrhkdBlPvrONjg7diF1EEl/SBz/ADRMHs3VfE29t3NN1ZxGRPk7BT+TOXDkZqfz27a1hlyIi0uMU/EBORhozzhnAM0u309x2KOxyRER6VDT33H3QzGrN7Lg3Sjezy8ys3syWBI9/7jRvhpmtMrO1ZvbNWBYeazdMHEJjSzt/XLEz7FJERHpUNHv8DwEzuujzirufFzzuAjCzVOAe4GpgHHCLmY07nWJ70tRR/SgvyNThHhFJeF0Gv7svAE7lrOcUYK27r3f3VuBR4PpT+Dm9IjXF+Oh5g/nT6jp2728JuxwRkR4Tq2P8U83sXTP7g5mdHbQNBmo69dkStMWtGyYNob3DeXKJvskrIokrFsH/NjDM3ScA/wU8eSo/xMxmm1mVmVXV1dXFoKzuGzsgnwlDCnlsUQ3uGtMvIonptIPf3RvcfX/w+hkg3cxKga1ARaeuQ4K2E/2cue5e6e6VZWVlp1vWKbvp/KGs2tnIkpp9odUgItKTTjv4zWyAmVnwekrwM3cDi4DRZjbCzDKAm4F5p/v7etpHJgwkOz2Vx6pquu4sItIHRTOc8xHgDWCsmW0xs1lmNsfM5gRdbgSWmdm7wE+Amz2iHfgCMB9YATzm7tU9sxixk5+VzofPHci8Jds40NIedjkiIjGX1lUHd7+li/l3A3efYN4zwDOnVlp4bjq/gt8s3sLTS7fzicqKrt8gItKH6Ju7x1E5rJiRZbk8tkiHe0Qk8Sj4j8PMuKmygqpNe3WdfhFJOAr+E7hh0hDSUoxfLdocdikiIjGl4D+BsvxMpp9dzq8Xb9GF20QkoSj4T+LWC4ex72AbT72nm7GLSOJQ8J/E1JH9OKN/Hg+/sTHsUkREYkbBfxJmxm0XDuPdLfW8q2/yikiCUPB34YZJg8nJSOXhhZvCLkVEJCYU/F3Iz0rnYxMH8/t3t7H3QGvY5YiInDYFfxRumzqMlvYOfr1YX+gSkb5PwR+FMwcUMGV4Cf9/4WY6OnS5ZhHp2xT8Ubpt6jA27znIS6tqwy5FROS0KPijNOOcAQwoyOKBVzeEXYqIyGlR8EcpPTWF2y8ezuvrdlO9rT7sckRETpmCvxtumTKUnIxU7n9Fe/0i0ncp+LuhMDudT1RW8Pt3t7GjvjnsckRETomCv5tmXTKCDnceen1j2KWIiJwSBX83VZTkMOOcAfzvm5t0a0YR6ZMU/Kdg1iUjaWhu1w3ZRaRPiuZm6w+aWa2ZLTvB/E+Z2XtmttTMXjezCZ3mbQzal5hZVSwLD9PkYcVMGlrEg69toP1QR9jliIh0SzR7/A8BM04yfwMwzd3HA98G5h4z/3J3P8/dK0+txPj0V9NGUbOnSdfqF5E+p8vgd/cFwJ6TzH/d3fcGkwuBITGqLa5deVY5Y8rzuOeltbqMg4j0KbE+xj8L+EOnaQeeM7PFZjb7ZG80s9lmVmVmVXV1dTEuK/ZSUozPX34Ga2r389zyHWGXIyIStZgFv5ldTiT4/65T8yXuPgm4Gvi8mV16ove7+1x3r3T3yrKysliV1aOuPXcQw/vlcPdLa3HXXr+I9A0xCX4zOxe4H7je3Xcfbnf3rcFzLfBbYEosfl+8SE0xPnfZGSzb2sDLq+P/U4qICMQg+M1sKPAEcJu7r+7Unmtm+YdfA9OB444M6ss+OnEwgwqzuOdF7fWLSN8QzXDOR4A3gLFmtsXMZpnZHDObE3T5Z6AfcO8xwzbLgVfN7F3gLeBpd3+2B5YhVBlpKcy5bBRVm/by5oYTngMXEYkbFo97qZWVlV5V1XeG/Te3HeID332JUWW5PDp7atjliEgSMrPF0Q6b1zd3YyArPZXPXTaKhev38NraXWGXIyJyUgr+GPnkBUMZVJjF9+av0rF+EYlrCv4YyUxL5YsfGs2Smn28uFK3ZxSR+KXgj6EbJw9hWL8c/vO51fo2r4jELQV/DKWnpvCVK0azYnsDzyzTNXxEJD4p+GPsugmDGd0/jx88v1pX7hSRuKTgj7HUFOPr08ewvu4Aj7+9JexyRETeR8HfA646ewAThxbx/edW6y5dIhJ3FPw9wMz4xw+fRW1jC3MXrA+7HBGRoyj4e8jkYSV8ePxA5i5Yz86G5rDLERE5QsHfg74xYyztHR18/7lVYZciInKEgr8HDeuXy8ypw/n14i2s2N4QdjkiIoCCv8d98YOjKchK5/89vUKXchCRuKDg72GFOel85YrRvLp2F/Ord4ZdjoiIgr833HbhMM4ckM+3n1pOU+uhsMsRkSSn4O8FaakpfOu6s9m6r4l7XlobdjkikuQU/L3kgpH9+Oh5g5i7YD0bdh0IuxwRSWIK/l7099ecRUZaCt/6fbVO9IpIaKIKfjN70Mxqzey4N0u3iJ+Y2Voze8/MJnWaN9PM1gSPmbEqvC/qX5DFV64Yzcur6nh+uU70ikg4ot3jfwiYcZL5VwOjg8ds4D4AMysB7gQuAKYAd5pZ8akWmwhmXjScseX53DmvmsbmtrDLEZEkFFXwu/sCYM9JulwP/MIjFgJFZjYQuAp43t33uPte4HlOvgFJeOmpKfz7x8ezo6GZ783XN3pFpPfF6hj/YKCm0/SWoO1E7Ult0tBibr9oOA8v3ETVxpNtT0VEYi9uTu6a2WwzqzKzqrq6urDL6XF/M30sgwqz+bvH36OlXWP7RaT3xCr4twIVnaaHBG0nan8fd5/r7pXuXllWVhajsuJXbmYa/3bDeNbVHeCeFzW2X0R6T6yCfx7w6WB0z4VAvbtvB+YD082sODipOz1oE2DamDJumDiYe19ep4u4iUiviXY45yPAG8BYM9tiZrPMbI6ZzQm6PAOsB9YC/wN8DsDd9wDfBhYFj7uCNgn807XjKMrJ4Ku/WqJDPiLSKywev0hUWVnpVVVVYZfRa15cuZPPPlTFnGmj+ObVZ4Zdjoj0QWa22N0ro+kbNyd3k9kHzyznlikV/HTBOhZplI+I9DAFf5z4xw+Po6I4h689toT9ukG7iPQgBX+cyM1M4/ufmMCWvU3861PLwy5HRBKYgj+OnD+8hDnTRvHoohqeem9b2OWISIJS8MeZr105holDi7jj8aVs3n0w7HJEJAEp+ONMemoK/3XLRMzgC4+8TWt7R9gliUiCUfDHoSHFOXz3xgm8t6We/3h2ZdjliEiCUfDHqRnnDOD2i4bzwKsbdO1+EYkpBX8cu+OaMxk/uJCv/WoJ6+v2h12OiCQIBX8cy0xL5b5bJ5GelsLshxdrfL+IxISCP84NKc7h7k9OZMOuA3z9sSV0dMTfJTZEpG9R8PcBF40q5Y6rz2R+9U7ufVmXcBaR06Pg7yNmXTKC688bxPefX80LK3SyV0ROnYK/jzAzvnPDuZw9qIAvPfIO1dvqwy5JRPooBX8fkp2RygMzz6cgO51ZD1Wxo7457JJEpA9S8Pcx5QVZPDDzfBqb2/jsQ4s4oJE+ItJNCv4+aNygAu751CRW7Wzki4+8wyGN9BGRblDw91GXje3Pv1x3Ni+urOUfn1xGPN5JTUTiU1o0ncxsBvBjIBW4392/c8z8HwKXB5M5QH93LwrmHQKWBvM2u/t1sShc4LYLh7F9XxP3vryO4px0vjFDt20Uka51GfxmlgrcA1wJbAEWmdk8dz9ytxB3/2qn/l8EJnb6EU3ufl7sSpbO/vaqsexrauPel9dRlJPO7EtHhV2SiMS5aPb4pwBr3X09gJk9ClwPnOg2UbcAd8amPOmKmfHt68+hoamNf3tmJYXZ6dx0/tCwyxKROBbNMf7BQE2n6S1B2/uY2TBgBPBip+YsM6sys4Vm9tFTrlROKDXF+MEnzmPamDLueGKp7t4lIicV65O7NwO/cfdDndqGuXsl8EngR2Z23GMRZjY72EBU1dXVxbisxJeRlsJ/3zqZymElfPnRJQp/ETmhaIJ/K1DRaXpI0HY8NwOPdG5w963B83rgZY4+/t+531x3r3T3yrKysijKkmNlZ6Tys8+cz+ShxQp/ETmhaIJ/ETDazEaYWQaRcJ93bCczOxMoBt7o1FZsZpnB61LgYk58bkBiIDczTeEvIifVZfC7ezvwBWA+sAJ4zN2rzewuM+s8NPNm4FE/ekD5WUCVmb0LvAR8p/NoIOkZx4b/k++c6AOaiCQji8cv/lRWVnpVVVXYZfR5B1ramfXzRSxcv4dvXXc2My8aHnZJItJDzGxxcD61S/rmbgLLzUzjoc9M4cpx5dw5r5of/XG1vuErIgr+RJeVnsp9n5rEjZOH8KM/ruFf5lXrLl4iSS6qSzZI35aWmsJ3P34uRdnp3P/qBnYfaOU//2ICWempYZcmIiFQ8CeJlBTjHz58FqX5mXznDyvZtq+J//l0Jf3yMsMuTUR6mQ71JBEzY860Udz7qUlUb2vgo/e+xtraxrDLEpFepuBPQteMH8iv/moqTa0dfOze13lt7a6wSxKRXqTgT1LnVRTx5OcvYmBhFp9+8C3uf2W9RvyIJAkFfxIbUpzD4399EVeeVc6/Pr2CLz26hIOtupWjSKJT8Ce5/Kx07rt1Et+YMZan39vGx+55nY27DoRdloj0IAW/YGZ87rIz+Plnp7CzsZmP3P0qzyzdHnZZItJDFPxyxAdGl/H7L1zCyLI8PvfLt7njiaU0tR7q+o0i0qco+OUoFSU5/GbOVOZMG8Ujb23mI3e/yortDWGXJSIxpOCX90lPTeGbV5/Jw7OmUN/UxvX3vMbPXtugSz2IJAgFv5zQB0aX8Ycvf4CLR/XjW79fzifvX8jm3QfDLktETpOCX06qNC+TB28/n//4+HiqtzYw48cL+MUbG7X3L9KHKfilS2bGTecPZf5XL2XysGL++XfVfOr+NzXsU6SPUvBL1AYVZfOLz07hOzeMZ9nWeqb/aAE//uMaWto18kekL1HwS7eYGTdPGcofvz6N6ePK+eEfVzPjR6/w6hpd70ekr1DwyykpL8ji7k9O4uFZU3B3bn3gTb74yDts29cUdmki0oWogt/MZpjZKjNba2bfPM78282szsyWBI+/7DRvppmtCR4zY1m8hO8Do8t49iuX8pUrRjO/egcf/P7L/OC5VRxo0TV/ROJVlzdbN7NUYDVwJbAFWATc4u7LO/W5Hah09y8c894SoAqoBBxYDEx2970n+5262XrfVLPnIN+bv4p5726jLD+Tv5k+hhsnV5CaYmGXJpLwYn2z9SnAWndf7+6twKPA9VHWchXwvLvvCcL+eWBGlO+VPqaiJIef3DKRJz53ERXF2fzd40v58E9e4aWVtbrks0gciSb4BwM1naa3BG3H+riZvWdmvzGzim6+FzObbWZVZlZVV1cXRVkSryYNLebxv76Iez45iYOth/jMQ4u44b7XeXXNLm0AROJArE7u/h4Y7u7nEtmr/3l3f4C7z3X3SnevLCsri1FZEhYz48PnDuSFr0/j328Yz876Zm594E1umruQhet3h12eSFKLJvi3AhWdpocEbUe4+253bwkm7wcmR/teSWzpqSncMmUoL/3tZdx1/dls3HWAm+cu5KafvsHLq3QISCQM0QT/ImC0mY0wswzgZmBe5w5mNrDT5HXAiuD1fGC6mRWbWTEwPWiTJJOZlsqnpw5nwTcu55+uHcem3Qe5/WeLuOYnr/K7JVtpP9QRdokiSaPL4Hf3duALRAJ7BfCYu1eb2V1mdl3Q7UtmVm1m7wJfAm4P3rsH+DaRjcci4K6gTZJUVnoqsy4ZwYJvXM73bjyX1vZDfPnRJVz+/Zd5+I2NGgYq0gu6HM4ZBg3nTB4dHc7zK3Zy38vrWFKzj/ysNP5icgW3TR3GiNLcsMsT6TO6M5xTwS9xwd15e/M+fv76Rp5Zup32DueysWXMnDqcaWPKSNF3AUROSsEvfVptQzP/+9ZmfvnmZuoaW6goyeYvJldw4+QhDCrKDrs8kbik4JeE0NrewbPVO3j0rc28vm43ZpFLRHyicghXjisnMy017BJF4oaCXxJOzZ6D/HrxFn5TVcO2+maKctK5bsIgrj9vEJOGFmOmQ0GS3BT8krAOdTivrd3Fr6pqeH75TlrbOxhclM21EwZy3YRBjBtYoI2AJCUFvySFxuY2nl++k9+/u41X1uyivcMZWZbLR84dxNXjBzC2PF8bAUkaCn5JOnsOtPLssh3Me3crb27YgztUlGQzfdwApo8rZ/KwYtJSdfsJSVwKfklqtY3NvLCilueqd/Da2t20HuqgOCedD51VzpXjyrn4jFLyMtPCLlMkphT8IoH9Le0sWF3Hc9U7eHFlLQ3N7aSlGJOGFTNtTBnTxpQxbmCBvicgfZ6CX+Q42g51ULVxLwvW1LFgdR3V2xoAKM3L4JIzSrl0TBkXjSplQGFWyJWKdJ+CXyQKdY0tvBJsBF5Zs4vdB1oBGN4vhwtH9uOCkSVcOLIfAwv1pTGJfwp+kW7q6HCWb29g4frdLFy/h7c27KahOXLBuGH9crhgRAlTRvRj0tAiRpTmarSQxB0Fv8hpOtThrNzRwML1e1i4fjdvbdhDfVMbAMU56UwcWszEiiImDStmQkWRThZL6LoT/PrXKnIcqSnG2YMKOXtQIbMuGUFHh7Omdj/vbN7L25v38vbmfby4shYAMxhbns/EoUWMH1zEOYMLGFOeT1a6Likh8Ul7/CKnqL6pjSU1+3h7017eqdnHks17jxweSksxRpfnc86gAs4ZXMg5gws4a2ABORna15KeoUM9IiFwd7bsbWLZ1nqWbatn2dYGlm2tP3LS2AxGluZy5sACxpbnM6Y8n7ED8hlakkOqhpPKadKhHpEQmBkVJTlUlORw9fjI3UjdnZ0NLUdtDJZuqefp97YfeV9mWgqjy/MiG4LyfMYMiGwUBhZk6fsF0iMU/CI9yMwYUJjFgMIsrhhXfqT9QEs7a2v3s2pnI6t3NLJqZyOvrd3FE29vPdInKz2F4f1yGVWWx4jSXEaW5QbPeRRmp4exOJIgogp+M5sB/BhIBe539+8cM/9rwF8C7UAd8Fl33xTMOwQsDbpudvfrEElyuZlpTKgoYkJF0VHt9QfbWF3byOqdjayvO8CGXQeo3lbPs9U7ONTx58Oy/XIzjmwIhpfmMrQkh4riHIaW5FCUk67hpnJSXQa/maUC9wBXAluARWY2z92Xd+r2DlDp7gfN7K+B7wI3BfOa3P28GNctkpAKc9I5f3gJ5w8vOaq9tb2DzXsOsr5uPxt2HTiyUXhxZS279rce1TcvM42KkhyGlmRHNgb9IoefKopzGFKcrdFGEtUe/xRgrbuvBzCzR4HrgSPB7+4vdeq/ELg1lkWKJLuMtBTO6J/HGf3z3jdvf0s7NXsOUrPnIJuD55q9TayrO8DLq+poae84qn9ZfiaDCrMYWJjNwKIsBhdlH3k9qDCbsvxMnWxOcNEE/2CgptP0FuCCk/SfBfyh03SWmVUROQz0HXd/sttVisgJ5WWmcdbAyHDRY7k7dY0t1OyNbBQ2725i274mttU3sbZuPwvW1HGw9dBR70lLMcoLshhU9OeNw4CCLMoLsuifn0n//Cz6F2Tqk0MfFtOTu2Z2K1AJTOvUPMzdt5rZSOBFM1vq7uuO897ZwGyAoUOHxrIskaRlZvQvyKJ/QRaTh5W8b76709DUzrb6JrbXN7FtXzPb9jWxvT7yvKRmH88ua6b1UMf73luQlRb52fmZRzYKZfmZ9C/Iojx47p+fSa6+1Rx3olkjW4GKTtNDgrajmNkVwD8A09y95XC7u28Nnteb2cvAROB9we/uc4G5EBnHH/0iiMipMjMKc9IpzEk/7icGiFzHaO/BVmobW9jZ0ExtYwt1jS3UNjSzs6GF2sZmFm3cQ21jC63t799A5GSk0i8vg5LcTEpzM+iXl0G/vEz6HX6dm0lJbgaleZHnjDTdMKenRRP8i4DRZjaCSODfDHyycwczmwj8FJjh7rWd2ouBg+7eYmalwMVETvyKSB+RkmKRoM7LPOHGASKfHuqb2qhtbKE22CDsbGhh9/4Wdh9oZdf+FnY0NFO9rYHdB1poO3T8/buCrLSjNgwluZmU5KZTlJ1BUU46xTkZFOemU5idQXFOOoXZ6bq7Wjd1Gfzu3m5mXwDmExnO+aC7V5vZXUCVu88DvgfkAb8OhpEdHrZ5FvBTM+sAUogc419+3F8kIn2amVGUk0FRTgZjyvNP2tfdaWhuZ/f+FvYcaGXX/lZ2H2hh9/7WYDryesOuA1Rt3Mu+prajhrMeKz8rLbJByEkPaohsIIpy0inKTqc4N1LX4Q1FQVY6+VlpSbvB0CUbRCTuuTuNLe3sO9DG3oOt7D3YSn1TG3sPtLL3YBv7Drayr6ntyOu9B1vZd7CNxuDaSSeSm5FKQfafNwYF2WnBczoFWWnB89Hth/vmZaXF1egnXbJBRBKKmUWCNyudof1yon5f26EO6puCDcPByIahvqmNhqY2GprbaGhqD54j09v2NbOyuZGGpjYaW9rpar84PzOyccjPSjvyKSIvK428zMhzfubh1+nkZaaRnxV5/Hl+OlnpKb3+hTsFv4gkrPTUFErzMinNy+z2ezs6nP2t7ZGNwlEbiPYTbjx2Njazrq6d/S3tNDS3H/dk97FSUyyyIchMY3BRNo/NmXoqi9otCn4RkeNISfnzpwyKT+1ntLQf4kDLIfY3t9PY0sb+5shGobG5ncaW9mC6LZjfTmYvjWhS8IuI9JDMtFQy01LijcE4AAAEgUlEQVQpyc0Iu5SjJOcpbRGRJKbgFxFJMgp+EZEko+AXEUkyCn4RkSSj4BcRSTIKfhGRJKPgFxFJMnF5kTYzqwM2neLbS4FdMSynL9AyJwctc+I7neUd5u5l0XSMy+A/HWZWFe0V6hKFljk5aJkTX28trw71iIgkGQW/iEiSScTgnxt2ASHQMicHLXPi65XlTbhj/CIicnKJuMcvIiInkTDBb2YzzGyVma01s2+GXU+smFmFmb1kZsvNrNrMvhy0l5jZ82a2JnguDtrNzH4S/B3eM7NJ4S7BqTOzVDN7x8yeCqZHmNmbwbL9yswygvbMYHptMH94mHWfKjMrMrPfmNlKM1thZlMTfT2b2VeDf9fLzOwRM8tKtPVsZg+aWa2ZLevU1u31amYzg/5rzGzm6dSUEMFvZqnAPcDVwDjgFjMbF25VMdMOfN3dxwEXAp8Plu2bwAvuPhp4IZiGyN9gdPCYDdzX+yXHzJeBFZ2m/wP4obufAewFZgXts4C9QfsPg3590Y+BZ939TGACkWVP2PVsZoOBLwGV7n4OkArcTOKt54eAGce0dWu9mlkJcCdwATAFuPPwxuKUuHuffwBTgfmdpu8A7gi7rh5a1t8BVwKrgIFB20BgVfD6p8Atnfof6deXHsCQ4D/EB4GnACPyxZa0Y9c5MB+YGrxOC/pZ2MvQzeUtBDYcW3cir2dgMFADlATr7SngqkRcz8BwYNmprlfgFuCnndqP6tfdR0Ls8fPnf0CHbQnaEkrw0XYi8CZQ7u7bg1k7gPLgdaL8LX4EfAM4fLfqfsA+d28Ppjsv15FlDubXB/37khFAHfCz4PDW/WaWSwKvZ3ffCvwnsBnYTmS9LSax1/Nh3V2vMV3fiRL8Cc/M8oDHga+4e0PneR7ZBUiY4Vlmdi1Q6+6Lw66lF6UBk4D73H0icIA/f/wHEnI9FwPXE9noDQJyef8hkYQXxnpNlODfClR0mh4StCUEM0snEvq/dPcnguadZjYwmD8QqA3aE+FvcTFwnZltBB4lcrjnx0CRmaUFfTov15FlDuYXArt7s+AY2AJscfc3g+nfENkQJPJ6vgLY4O517t4GPEFk3Sfyej6su+s1pus7UYJ/ETA6GA2QQeQE0byQa4oJMzPgAWCFu/+g06x5wOEz+zOJHPs/3P7pYHTAhUB9p4+UfYK73+HuQ9x9OJF1+aK7fwp4Cbgx6HbsMh/+W9wY9O9Te8buvgOoMbOxQdOHgOUk8HomcojnQjPLCf6dH17mhF3PnXR3vc4HpptZcfBJaXrQdmrCPukRw5Mn1wCrgXXAP4RdTwyX6xIiHwPfA5YEj2uIHNt8AVgD/BEoCfobkRFO64ClREZMhL4cp7H8lwFPBa9HAm8Ba4FfA5lBe1YwvTaYPzLsuk9xWc8DqoJ1/SRQnOjrGfgWsBJYBjwMZCbaegYeIXIOo43IJ7tZp7Jegc8Gy74W+Mzp1KRv7oqIJJlEOdQjIiJRUvCLiCQZBb+ISJJR8IuIJBkFv4hIklHwi4gkGQW/iEiSUfCLiCSZ/wPDK19p8dG+OgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        gradient_w = (z-y)*x\n",
    "        gradient_w = np.mean(gradient_w, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = (z - y)\n",
    "        gradient_b = np.mean(gradient_b)        \n",
    "        return gradient_w, gradient_b\n",
    "    \n",
    "    def update(self, gradient_w, gradient_b, eta = 0.01):\n",
    "        self.w = self.w - eta * gradient_w\n",
    "        self.b = self.b - eta * gradient_b\n",
    "        \n",
    "    def train(self, x, y, iterations=100, eta=0.01):\n",
    "        losses = []\n",
    "        for i in range(iterations):\n",
    "            z = self.forward(x)\n",
    "            L = self.loss(z, y)\n",
    "            gradient_w, gradient_b = self.gradient(x, y)\n",
    "            self.update(gradient_w, gradient_b, eta)\n",
    "            losses.append(L)\n",
    "            if (i+1) % 10 == 0:\n",
    "                print('iter {}, loss {}'.format(i, L))\n",
    "        return losses\n",
    "\n",
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "x = train_data[:, :-1]\n",
    "y = train_data[:, -1:]\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "num_iterations=1000\n",
    "# 启动训练\n",
    "losses = net.train(x,y, iterations=num_iterations, eta=0.01)\n",
    "\n",
    "# 画出损失函数的变化趋势\n",
    "plot_x = np.arange(num_iterations)\n",
    "plot_y = np.array(losses)\n",
    "plt.plot(plot_x, plot_y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "### 小批量随机梯度下降法（Mini-batch Stochastic Gradient Descent）\n",
    "\n",
    "\n",
    "在上述程序中，每次迭代的时候均基于数据集中的全部数据进行计算。但在实际问题中数据集往往非常大，如果每次计算都使用全部的数据来计算损失函数和梯度，效率非常低。一个合理的解决方案是每次从总的数据集中随机抽取出小部分数据来代表整体，基于这部分数据计算梯度和损失，然后更新参数。这种方法被称作小批量随机梯度下降法（Mini-batch Stochastic Gradient Descent），简称SGD。每次迭代时抽取出来的一批数据被称为一个min-batch，一个mini-batch所包含的样本数目称为batch_size。当程序迭代的时候，按mini-batch逐渐抽取出样本，当把整个数据集都遍历到了的时候，则完成了一轮的训练，也叫一个epoch。启动训练时，可以将训练的轮数num_epochs和batch_size作为参数传入。\n",
    "\n",
    "下面结合程序介绍具体的实现过程。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(404, 14)"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "train_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "train_data中一共包含404条数据，如果batch_size=10，即取前0-9号样本作为第一个mini-batch，命名train_data1。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(10, 14)"
      ]
     },
     "execution_count": 38,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data1 = train_data[0:10]\n",
    "train_data1.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "使用train_data1的数据（0-9号样本）计算梯度并更新网络参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.9001866101467375]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "net = Network(13)\n",
    "x = train_data1[:, :-1]\n",
    "y = train_data1[:, -1:]\n",
    "loss = net.train(x, y, iterations=1, eta=0.01)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "再取出10-19号样本作为第二个mini-batch，计算梯度并更新网络参数。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[0.8903272433979657]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_data2 = train_data[10:19]\n",
    "x = train_data1[:, :-1]\n",
    "y = train_data1[:, -1:]\n",
    "loss = net.train(x, y, iterations=1, eta=0.01)\n",
    "loss"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "按此方法不断的取出新的mini-batch并逐渐更新网络参数。\n",
    "下面的程序可以将train_data分成大小为batch_size的多个mini_batch。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "total number of mini_batches is  41\n",
      "first mini_batch shape  (10, 14)\n",
      "last mini_batch shape  (4, 14)\n"
     ]
    }
   ],
   "source": [
    "batch_size = 10\n",
    "n = len(train_data)\n",
    "mini_batches = [train_data[k:k+batch_size] for k in range(0, n, batch_size)]\n",
    "print('total number of mini_batches is ', len(mini_batches))\n",
    "print('first mini_batch shape ', mini_batches[0].shape)\n",
    "print('last mini_batch shape ', mini_batches[-1].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "上面的代码将train_data分成 $\\frac{404}{10} + 1 = 41$ 个 mini_batch了，其中前40个mini_batch，每个均含有10个样本，最后一个mini_batch只含有4个样本。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "另外，我们这里是按顺序取出mini_batch的，而SGD里面是随机的抽取一部分样本代表总体。为了实现随机抽样的效果，我们先将train_data里面的样本顺序随机打乱，然后再抽取mini_batch。随机打乱样本顺序，需要用到np.random.shuffle函数，下面先介绍它的用法。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before shuffle [ 1  2  3  4  5  6  7  8  9 10 11 12]\n",
      "after shuffle [ 7  2 11  3  8  6 12  1  4  5 10  9]\n"
     ]
    }
   ],
   "source": [
    "# 新建一个array\n",
    "a = np.array([1,2,3,4,5,6,7,8,9,10,11,12])\n",
    "print('before shuffle', a)\n",
    "np.random.shuffle(a)\n",
    "print('after shuffle', a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "多次运行上面的代码，可以发现每次执行shuffle函数后的数字顺序均不同。\n",
    "上面举的是一个1维数组乱序的案例，我们在观察下2维数组乱序后的效果。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "before shuffle\n",
      " [[ 1  2]\n",
      " [ 3  4]\n",
      " [ 5  6]\n",
      " [ 7  8]\n",
      " [ 9 10]\n",
      " [11 12]]\n",
      "after shuffle\n",
      " [[ 1  2]\n",
      " [ 3  4]\n",
      " [ 5  6]\n",
      " [ 9 10]\n",
      " [11 12]\n",
      " [ 7  8]]\n"
     ]
    }
   ],
   "source": [
    "# 新建一个array\n",
    "a = np.array([1,2,3,4,5,6,7,8,9,10,11,12])\n",
    "a = a.reshape([6, 2])\n",
    "print('before shuffle\\n', a)\n",
    "np.random.shuffle(a)\n",
    "print('after shuffle\\n', a)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "观察运行结果可发现，数组的元素在第0维被随机打乱，但第1维的顺序保持不变。例如数字2仍然紧挨在数字1的后面，数字8仍然紧挨在数字7的后面，而第二维的[3, 4]并不排在[1, 2]的后面。"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "综上随机乱序和抽取mini_batch的步骤，我们可以改写训练过程如下。每个随机抽取的mini-batch数据，输入到模型中用于参数训练。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "\n",
    "# 打乱样本顺序\n",
    "np.random.shuffle(train_data)\n",
    "\n",
    "# 将train_data分成多个mini_batch\n",
    "batch_size = 10\n",
    "n = len(train_data)\n",
    "mini_batches = [train_data[k:k+batch_size] for k in range(0, n, batch_size)]\n",
    "\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "\n",
    "# 依次使用每个mini_batch的数据\n",
    "for mini_batch in mini_batches:\n",
    "    x = mini_batch[:, :-1]\n",
    "    y = mini_batch[:, -1:]\n",
    "    loss = net.train(x, y, iterations=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "将这部分实现SGD算法的代码集成到Network类中的train函数中，最终的完整代码如下。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch   0 / iter   0, loss = 0.6273\n",
      "Epoch   0 / iter   1, loss = 0.4835\n",
      "Epoch   0 / iter   2, loss = 0.5830\n",
      "Epoch   0 / iter   3, loss = 0.5466\n",
      "Epoch   0 / iter   4, loss = 0.2147\n",
      "Epoch   1 / iter   0, loss = 0.6645\n",
      "Epoch   1 / iter   1, loss = 0.4875\n",
      "Epoch   1 / iter   2, loss = 0.4707\n",
      "Epoch   1 / iter   3, loss = 0.4153\n",
      "Epoch   1 / iter   4, loss = 0.1402\n",
      "Epoch   2 / iter   0, loss = 0.5897\n",
      "Epoch   2 / iter   1, loss = 0.4373\n",
      "Epoch   2 / iter   2, loss = 0.4631\n",
      "Epoch   2 / iter   3, loss = 0.3960\n",
      "Epoch   2 / iter   4, loss = 0.2340\n",
      "Epoch   3 / iter   0, loss = 0.4139\n",
      "Epoch   3 / iter   1, loss = 0.5635\n",
      "Epoch   3 / iter   2, loss = 0.3807\n",
      "Epoch   3 / iter   3, loss = 0.3975\n",
      "Epoch   3 / iter   4, loss = 0.1207\n",
      "Epoch   4 / iter   0, loss = 0.3786\n",
      "Epoch   4 / iter   1, loss = 0.4474\n",
      "Epoch   4 / iter   2, loss = 0.4019\n",
      "Epoch   4 / iter   3, loss = 0.4352\n",
      "Epoch   4 / iter   4, loss = 0.0435\n",
      "Epoch   5 / iter   0, loss = 0.4387\n",
      "Epoch   5 / iter   1, loss = 0.3886\n",
      "Epoch   5 / iter   2, loss = 0.3182\n",
      "Epoch   5 / iter   3, loss = 0.4189\n",
      "Epoch   5 / iter   4, loss = 0.1741\n",
      "Epoch   6 / iter   0, loss = 0.3191\n",
      "Epoch   6 / iter   1, loss = 0.3601\n",
      "Epoch   6 / iter   2, loss = 0.4199\n",
      "Epoch   6 / iter   3, loss = 0.3289\n",
      "Epoch   6 / iter   4, loss = 1.2691\n",
      "Epoch   7 / iter   0, loss = 0.3202\n",
      "Epoch   7 / iter   1, loss = 0.2855\n",
      "Epoch   7 / iter   2, loss = 0.4129\n",
      "Epoch   7 / iter   3, loss = 0.3331\n",
      "Epoch   7 / iter   4, loss = 0.2218\n",
      "Epoch   8 / iter   0, loss = 0.2368\n",
      "Epoch   8 / iter   1, loss = 0.3457\n",
      "Epoch   8 / iter   2, loss = 0.3339\n",
      "Epoch   8 / iter   3, loss = 0.3812\n",
      "Epoch   8 / iter   4, loss = 0.0534\n",
      "Epoch   9 / iter   0, loss = 0.3567\n",
      "Epoch   9 / iter   1, loss = 0.4033\n",
      "Epoch   9 / iter   2, loss = 0.1926\n",
      "Epoch   9 / iter   3, loss = 0.2803\n",
      "Epoch   9 / iter   4, loss = 0.1557\n",
      "Epoch  10 / iter   0, loss = 0.3435\n",
      "Epoch  10 / iter   1, loss = 0.2790\n",
      "Epoch  10 / iter   2, loss = 0.3456\n",
      "Epoch  10 / iter   3, loss = 0.2076\n",
      "Epoch  10 / iter   4, loss = 0.0935\n",
      "Epoch  11 / iter   0, loss = 0.3024\n",
      "Epoch  11 / iter   1, loss = 0.2517\n",
      "Epoch  11 / iter   2, loss = 0.2797\n",
      "Epoch  11 / iter   3, loss = 0.2989\n",
      "Epoch  11 / iter   4, loss = 0.0301\n",
      "Epoch  12 / iter   0, loss = 0.2507\n",
      "Epoch  12 / iter   1, loss = 0.2563\n",
      "Epoch  12 / iter   2, loss = 0.2971\n",
      "Epoch  12 / iter   3, loss = 0.2833\n",
      "Epoch  12 / iter   4, loss = 0.0597\n",
      "Epoch  13 / iter   0, loss = 0.2827\n",
      "Epoch  13 / iter   1, loss = 0.2094\n",
      "Epoch  13 / iter   2, loss = 0.2417\n",
      "Epoch  13 / iter   3, loss = 0.2985\n",
      "Epoch  13 / iter   4, loss = 0.4036\n",
      "Epoch  14 / iter   0, loss = 0.3085\n",
      "Epoch  14 / iter   1, loss = 0.2015\n",
      "Epoch  14 / iter   2, loss = 0.1830\n",
      "Epoch  14 / iter   3, loss = 0.2978\n",
      "Epoch  14 / iter   4, loss = 0.0630\n",
      "Epoch  15 / iter   0, loss = 0.2342\n",
      "Epoch  15 / iter   1, loss = 0.2780\n",
      "Epoch  15 / iter   2, loss = 0.2571\n",
      "Epoch  15 / iter   3, loss = 0.1838\n",
      "Epoch  15 / iter   4, loss = 0.0627\n",
      "Epoch  16 / iter   0, loss = 0.1896\n",
      "Epoch  16 / iter   1, loss = 0.1966\n",
      "Epoch  16 / iter   2, loss = 0.2018\n",
      "Epoch  16 / iter   3, loss = 0.3257\n",
      "Epoch  16 / iter   4, loss = 0.1268\n",
      "Epoch  17 / iter   0, loss = 0.1990\n",
      "Epoch  17 / iter   1, loss = 0.2031\n",
      "Epoch  17 / iter   2, loss = 0.2662\n",
      "Epoch  17 / iter   3, loss = 0.2128\n",
      "Epoch  17 / iter   4, loss = 0.0133\n",
      "Epoch  18 / iter   0, loss = 0.1780\n",
      "Epoch  18 / iter   1, loss = 0.1575\n",
      "Epoch  18 / iter   2, loss = 0.2547\n",
      "Epoch  18 / iter   3, loss = 0.2544\n",
      "Epoch  18 / iter   4, loss = 0.2007\n",
      "Epoch  19 / iter   0, loss = 0.1657\n",
      "Epoch  19 / iter   1, loss = 0.2000\n",
      "Epoch  19 / iter   2, loss = 0.2045\n",
      "Epoch  19 / iter   3, loss = 0.2524\n",
      "Epoch  19 / iter   4, loss = 0.0632\n",
      "Epoch  20 / iter   0, loss = 0.1629\n",
      "Epoch  20 / iter   1, loss = 0.1895\n",
      "Epoch  20 / iter   2, loss = 0.2523\n",
      "Epoch  20 / iter   3, loss = 0.1896\n",
      "Epoch  20 / iter   4, loss = 0.0918\n",
      "Epoch  21 / iter   0, loss = 0.1583\n",
      "Epoch  21 / iter   1, loss = 0.2322\n",
      "Epoch  21 / iter   2, loss = 0.1567\n",
      "Epoch  21 / iter   3, loss = 0.2089\n",
      "Epoch  21 / iter   4, loss = 0.2035\n",
      "Epoch  22 / iter   0, loss = 0.2273\n",
      "Epoch  22 / iter   1, loss = 0.1427\n",
      "Epoch  22 / iter   2, loss = 0.1712\n",
      "Epoch  22 / iter   3, loss = 0.1826\n",
      "Epoch  22 / iter   4, loss = 0.2878\n",
      "Epoch  23 / iter   0, loss = 0.1685\n",
      "Epoch  23 / iter   1, loss = 0.1622\n",
      "Epoch  23 / iter   2, loss = 0.1499\n",
      "Epoch  23 / iter   3, loss = 0.2329\n",
      "Epoch  23 / iter   4, loss = 0.1486\n",
      "Epoch  24 / iter   0, loss = 0.1617\n",
      "Epoch  24 / iter   1, loss = 0.2083\n",
      "Epoch  24 / iter   2, loss = 0.1442\n",
      "Epoch  24 / iter   3, loss = 0.1740\n",
      "Epoch  24 / iter   4, loss = 0.1641\n",
      "Epoch  25 / iter   0, loss = 0.1159\n",
      "Epoch  25 / iter   1, loss = 0.2064\n",
      "Epoch  25 / iter   2, loss = 0.1690\n",
      "Epoch  25 / iter   3, loss = 0.1778\n",
      "Epoch  25 / iter   4, loss = 0.0159\n",
      "Epoch  26 / iter   0, loss = 0.1730\n",
      "Epoch  26 / iter   1, loss = 0.1861\n",
      "Epoch  26 / iter   2, loss = 0.1387\n",
      "Epoch  26 / iter   3, loss = 0.1486\n",
      "Epoch  26 / iter   4, loss = 0.1090\n",
      "Epoch  27 / iter   0, loss = 0.1393\n",
      "Epoch  27 / iter   1, loss = 0.1775\n",
      "Epoch  27 / iter   2, loss = 0.1564\n",
      "Epoch  27 / iter   3, loss = 0.1245\n",
      "Epoch  27 / iter   4, loss = 0.7611\n",
      "Epoch  28 / iter   0, loss = 0.1470\n",
      "Epoch  28 / iter   1, loss = 0.1211\n",
      "Epoch  28 / iter   2, loss = 0.1285\n",
      "Epoch  28 / iter   3, loss = 0.1854\n",
      "Epoch  28 / iter   4, loss = 0.5240\n",
      "Epoch  29 / iter   0, loss = 0.1740\n",
      "Epoch  29 / iter   1, loss = 0.0898\n",
      "Epoch  29 / iter   2, loss = 0.1392\n",
      "Epoch  29 / iter   3, loss = 0.1842\n",
      "Epoch  29 / iter   4, loss = 0.0251\n",
      "Epoch  30 / iter   0, loss = 0.0978\n",
      "Epoch  30 / iter   1, loss = 0.1529\n",
      "Epoch  30 / iter   2, loss = 0.1640\n",
      "Epoch  30 / iter   3, loss = 0.1503\n",
      "Epoch  30 / iter   4, loss = 0.0975\n",
      "Epoch  31 / iter   0, loss = 0.1399\n",
      "Epoch  31 / iter   1, loss = 0.1595\n",
      "Epoch  31 / iter   2, loss = 0.1209\n",
      "Epoch  31 / iter   3, loss = 0.1203\n",
      "Epoch  31 / iter   4, loss = 0.2008\n",
      "Epoch  32 / iter   0, loss = 0.1501\n",
      "Epoch  32 / iter   1, loss = 0.1310\n",
      "Epoch  32 / iter   2, loss = 0.1065\n",
      "Epoch  32 / iter   3, loss = 0.1489\n",
      "Epoch  32 / iter   4, loss = 0.0818\n",
      "Epoch  33 / iter   0, loss = 0.1401\n",
      "Epoch  33 / iter   1, loss = 0.1367\n",
      "Epoch  33 / iter   2, loss = 0.0970\n",
      "Epoch  33 / iter   3, loss = 0.1481\n",
      "Epoch  33 / iter   4, loss = 0.0711\n",
      "Epoch  34 / iter   0, loss = 0.1157\n",
      "Epoch  34 / iter   1, loss = 0.1050\n",
      "Epoch  34 / iter   2, loss = 0.1378\n",
      "Epoch  34 / iter   3, loss = 0.1505\n",
      "Epoch  34 / iter   4, loss = 0.0429\n",
      "Epoch  35 / iter   0, loss = 0.1096\n",
      "Epoch  35 / iter   1, loss = 0.1279\n",
      "Epoch  35 / iter   2, loss = 0.1715\n",
      "Epoch  35 / iter   3, loss = 0.0888\n",
      "Epoch  35 / iter   4, loss = 0.0473\n",
      "Epoch  36 / iter   0, loss = 0.1350\n",
      "Epoch  36 / iter   1, loss = 0.0781\n",
      "Epoch  36 / iter   2, loss = 0.1458\n",
      "Epoch  36 / iter   3, loss = 0.1288\n",
      "Epoch  36 / iter   4, loss = 0.0421\n",
      "Epoch  37 / iter   0, loss = 0.1083\n",
      "Epoch  37 / iter   1, loss = 0.0972\n",
      "Epoch  37 / iter   2, loss = 0.1513\n",
      "Epoch  37 / iter   3, loss = 0.1236\n",
      "Epoch  37 / iter   4, loss = 0.0366\n",
      "Epoch  38 / iter   0, loss = 0.1204\n",
      "Epoch  38 / iter   1, loss = 0.1341\n",
      "Epoch  38 / iter   2, loss = 0.1109\n",
      "Epoch  38 / iter   3, loss = 0.0905\n",
      "Epoch  38 / iter   4, loss = 0.3906\n",
      "Epoch  39 / iter   0, loss = 0.0923\n",
      "Epoch  39 / iter   1, loss = 0.1094\n",
      "Epoch  39 / iter   2, loss = 0.1295\n",
      "Epoch  39 / iter   3, loss = 0.1239\n",
      "Epoch  39 / iter   4, loss = 0.0684\n",
      "Epoch  40 / iter   0, loss = 0.1188\n",
      "Epoch  40 / iter   1, loss = 0.0984\n",
      "Epoch  40 / iter   2, loss = 0.1067\n",
      "Epoch  40 / iter   3, loss = 0.1057\n",
      "Epoch  40 / iter   4, loss = 0.4602\n",
      "Epoch  41 / iter   0, loss = 0.1478\n",
      "Epoch  41 / iter   1, loss = 0.0980\n",
      "Epoch  41 / iter   2, loss = 0.0921\n",
      "Epoch  41 / iter   3, loss = 0.1020\n",
      "Epoch  41 / iter   4, loss = 0.0430\n",
      "Epoch  42 / iter   0, loss = 0.0991\n",
      "Epoch  42 / iter   1, loss = 0.0994\n",
      "Epoch  42 / iter   2, loss = 0.1270\n",
      "Epoch  42 / iter   3, loss = 0.0988\n",
      "Epoch  42 / iter   4, loss = 0.1176\n",
      "Epoch  43 / iter   0, loss = 0.1286\n",
      "Epoch  43 / iter   1, loss = 0.1013\n",
      "Epoch  43 / iter   2, loss = 0.1066\n",
      "Epoch  43 / iter   3, loss = 0.0779\n",
      "Epoch  43 / iter   4, loss = 0.1481\n",
      "Epoch  44 / iter   0, loss = 0.0840\n",
      "Epoch  44 / iter   1, loss = 0.0858\n",
      "Epoch  44 / iter   2, loss = 0.1388\n",
      "Epoch  44 / iter   3, loss = 0.1000\n",
      "Epoch  44 / iter   4, loss = 0.0313\n",
      "Epoch  45 / iter   0, loss = 0.0896\n",
      "Epoch  45 / iter   1, loss = 0.1173\n",
      "Epoch  45 / iter   2, loss = 0.0916\n",
      "Epoch  45 / iter   3, loss = 0.1043\n",
      "Epoch  45 / iter   4, loss = 0.0074\n",
      "Epoch  46 / iter   0, loss = 0.1008\n",
      "Epoch  46 / iter   1, loss = 0.0915\n",
      "Epoch  46 / iter   2, loss = 0.0877\n",
      "Epoch  46 / iter   3, loss = 0.1139\n",
      "Epoch  46 / iter   4, loss = 0.0292\n",
      "Epoch  47 / iter   0, loss = 0.0679\n",
      "Epoch  47 / iter   1, loss = 0.0987\n",
      "Epoch  47 / iter   2, loss = 0.0929\n",
      "Epoch  47 / iter   3, loss = 0.1098\n",
      "Epoch  47 / iter   4, loss = 0.4838\n",
      "Epoch  48 / iter   0, loss = 0.0693\n",
      "Epoch  48 / iter   1, loss = 0.1095\n",
      "Epoch  48 / iter   2, loss = 0.1128\n",
      "Epoch  48 / iter   3, loss = 0.0890\n",
      "Epoch  48 / iter   4, loss = 0.1008\n",
      "Epoch  49 / iter   0, loss = 0.0724\n",
      "Epoch  49 / iter   1, loss = 0.0804\n",
      "Epoch  49 / iter   2, loss = 0.0919\n",
      "Epoch  49 / iter   3, loss = 0.1233\n",
      "Epoch  49 / iter   4, loss = 0.1849\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4zLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvIxREBQAAIABJREFUeJztvXmYHFd19/89Xb33TM8uaUbbaLdlW97k3XgDjG1ebOAFYp4AgZg48MOENywJ2YAYCD9CwpuQOGAHHAIEO2YJFtiOTbzJuy3bkmXtI2kkzaLZt96Xuu8fVfd2VS+aHk33dPf0+TyPHnVX13Tf6uVbp77n3HNJCAGGYRhmceGo9AAYhmGY0sPizjAMswhhcWcYhlmEsLgzDMMsQljcGYZhFiEs7gzDMIsQFneGYZhFCIs7wzDMIoTFnWEYZhHirNQLt7e3i+7u7kq9PMMwTE3y6quvjgohOmbbr2Li3t3djR07dlTq5RmGYWoSIjpWzH5syzAMwyxCWNwZhmEWISzuDMMwixAWd4ZhmEUIizvDMMwihMWdYRhmEcLizjAMswhhcS8TO09M4s3+qUoPg2GYOoXFvUx8/aG9+NajByo9DIZh6hQW9zKRSAukdL3Sw2AYpk5hcS8TQgiwtjMMUylY3MuELgR0ISo9DIZh6hQW9zKR1sHizjBMxWBxLxNCCOis7QzDVAgW9zLBtgzDMJVkVnEnonuJaJiI3izw+O8S0RtEtJuInieic0s/zNpDF+DInWGYilFM5P5DADec4vGjAK4WQpwD4KsA7inBuGoeXQgIjtwZhqkQs67EJITYTkTdp3j8ecvdFwGsmP+wah8hOKHKMEzlKLXnfhuAR0r8nDWJznXuDMNUkJKtoUpE18IQ9ytPsc/tAG4HgFWrVpXqpasSTqgyDFNJShK5E9EWAN8HcIsQYqzQfkKIe4QQW4UQWzs6Zl28u6bRdcOaYRiGqQTzFnciWgXglwA+LIQ4OP8hLQ44cmcYppLMassQ0X0ArgHQTkR9AL4MwAUAQojvAfgSgDYA/0JEAJASQmwt14BrBV0IpFncGYapEMVUy3xwlsc/DuDjJRvRIkEXbMswDFM5eIZqmRBsyzAMU0FY3MuEznXuDMNUEBb3MsF17gzDVBIW9zKh69x+gGGYysHiXia4cRjDMJWExb1McJ07wzCVhMW9TOi8WAfDMBWExb1McLUMwzCVhMW9THCdO8MwlYTFvUzowqiYYRiGqQQs7mXCWImp0qNgGKZeYXEvA8IUdrZlGIapFCzuZUC6MezKMAxTKVjcy4CM2DlyZximUrC4lwEp6qztDMNUChb3MiCULcPqzjBMZWBxLwNS1HklJoZhKgWLexmQiVQhwJ0hGYapCCzuZcBqx7C2MwxTCVjcy4CwLNLBvjvDMJWAxb0MWL12rnVnGKYSsLiXAd0m7qzuDMMsPCzuZYA9d4ZhKs2s4k5E9xLRMBG9WeBxIqLvEFEPEb1BRBeUfpi1hVXQOXJnGKYSFBO5/xDADad4/EYAG8x/twP47vyHVduwLcMwTKWZVdyFENsBjJ9il1sA/EgYvAigmYg6SzXAWkQX+W8zDMMsFKXw3JcDOGG532duy4GIbieiHUS0Y2RkpAQvXZ1YF+ngBTsYhqkEC5pQFULcI4TYKoTY2tHRsZAvvaCw584wTKUphbj3A1hpub/C3Fa3cJ07wzCVphTivg3AR8yqmUsBTAkhBkvwvDWLvRSS1Z1hmIXHOdsORHQfgGsAtBNRH4AvA3ABgBDiewAeBnATgB4AEQAfK9dgawXBkTvDMBVmVnEXQnxwlscFgE+VbESLAJ09d4ZhKgzPUC0DXOfOMEylYXEvA7qlKyRrO8MwlYDFvQxw5M4wTKVhcS8DVj1Pc0aVYZgKwOJeBrjOnWGYSsPiXga4zp1hmErD4l4GuM6dYZhKw+JeBrjOnWGYSsPiXgZsXSFZ3BmGqQAs7mXAGrmztjMMUwlY3MuA4Dp3hmEqDIt7GeCVmBiGqTQs7mXAWufOk5gYhqkELO5lgOvcGYapNCzuZYDr3BmGqTQs7mXA2hWSE6oMw1QCFvcywF0hGYapNCzuZYDr3BmGqTQs7mWA69wZhqk0LO5lgOvcGYapNCzuZSDNkTvDMBWGxb0MCK5zZximwhQl7kR0AxEdIKIeIvpinsdXEdGTRPQ6Eb1BRDeVfqi1g26boVrBgTAMU7fMKu5EpAG4C8CNADYD+CARbc7a7S8BPCCEOB/ArQD+pdQDrSW4zp1hmEpTTOR+MYAeIcQRIUQCwP0AbsnaRwAImrebAAyUboi1B7cfYBim0jiL2Gc5gBOW+30ALsna5ysAHiOiTwMIAHhbSUZXowiulmEYpsKUKqH6QQA/FEKsAHATgB8TUc5zE9HtRLSDiHaMjIyU6KWrD56hyjBMpSlG3PsBrLTcX2Fus3IbgAcAQAjxAgAvgPbsJxJC3COE2CqE2NrR0XF6I64BuM6dYZhKU4y4vwJgAxGtISI3jITptqx9jgN4KwAQ0ZkwxH3xhuazkGbPnWGYCjOruAshUgDuAPAogH0wqmL2ENGdRHSzudvnAPwBEe0CcB+Aj4o6VjVuP8AwTKUpJqEKIcTDAB7O2vYly+29AK4o7dBqF93ixehc584wTAXgGaplwOqzpzlyr0te6R3HB+5+AUmexcZUCBb3MsB17swbfVN4+eg4pqPJSg+FqVNY3MsA17kz0prjKzemUrC4lwGuc2ekqKf57M5UCBb3MsB17owUdRZ3plKwuJcB9twZKepcLcVUChb3MmAvhWRxr0fS7LkzFYbFvQywLcPoynPn0J2pDCzuZYATqkzGc6/wQJi6hcW9DNiX2avgQJiKwdUyTKVhcS8DPEOVkbkWvnJjKgWLexlgW4aRdkyKI3emQrC4lwFdAETGbdb2+kQmUtmWYSoFi3sZ0IWAy2G8tVwKWZ9IO46v3JhKweJeBnRdQHMYoTtre30ibRmO3JlKsWjEfSaWxKf+4zWMheKVHgp0AYu484+7HtG5/QBTYRaNuB84OYOHdg/i9eOTlR4KdCHgIMN35/YD9QmXQjKVZtGIezJt/IjCiVSFR2IIusNBcBCxLVOncMtfptIsGnGXEVI0ka7wSAxbxkEEB7EtU6+ohCqf3ZkKsWjEPWWWnkWqQtylLUMcudUpsr6d69yZSrF4xN20ZSJVYMsYde5G5M7aXp+oGaos7kyFqDlxf2T3IDb+5SM4PBKybZcRUlVE7rqARqbnzj/uuoRb/jKVpihxJ6IbiOgAEfUQ0RcL7PMBItpLRHuI6KelHWYGt9OBREpHOG6P0KvRluGEav2ic7UMU2Gcs+1ARBqAuwC8HUAfgFeIaJsQYq9lnw0A/gzAFUKICSJaUq4BN3iMIYdidnGvtoQqcUK1ruFl9phKU0zkfjGAHiHEESFEAsD9AG7J2ucPANwlhJgAACHEcGmHmaHBa4j7TFbkXn2lkIDDQVznXqeYX0cWd6ZiFCPuywGcsNzvM7dZ2QhgIxE9R0QvEtENpRpgNo0eF4B8kbthy1RH5C7MUki2ZeoVbvnLVJpZbZk5PM8GANcAWAFgOxGdI4SwTRclotsB3A4Aq1atOq0XkpF7qEDkXh2eO9e51zsyB8SlkEylKCZy7wew0nJ/hbnNSh+AbUKIpBDiKICDMMTehhDiHiHEViHE1o6OjtMacMCjAcgV97SevxQyFE/hwq/+Fs8eGj2t1zsddCFAZp07/7brE7l0KldLMZWiGHF/BcAGIlpDRG4AtwLYlrXPr2BE7SCidhg2zZESjlPhcWpwaw7MxFL4yL0v464newAULoWcCCcwFk7g4NBMOYaTF2GN3PnHXZdwbxmm0swq7kKIFIA7ADwKYB+AB4QQe4joTiK62dztUQBjRLQXwJMAviCEGCvXoBu8ToTiSezoHcd/v3kSAJBK5y+FlKKfXTpZTtLWOne2ZeqSTJ17hQfC1C1Fee5CiIcBPJy17UuW2wLAZ81/ZafB48ToTAKRRBr7BqcRS6YtkXv+RGtoAatopC3DCdX6JVPnrld4JEy9UnMzVAFD3E9MRAAYkfmegSlL+wF75K5KJAtE7l/7zV58+7EDJR2fTKhyy9/6JVPnXuGBMHVLqaplFpQGrxP7BqfV/Z0nplSEFE/phi1iLpYhf2TZpZOSZ3tGEfS6Sjo+WeeuOdiWqVfSXArJVJiajNwbPU7MWMR654lJJC3+h9WaSZqhUyiev0QynEghkiytZcN17gy3H2AqTU2Ku6x1B4DuNj8OnJy2/YisE5nSsyRUw/F0yWvjZfsB4jr3uoVb/jKVpibFPeDJiPvKVj/iKV1F6IDdd1fVMgUSqqFYquSzWq2Nw1jb6xNu+ctUmpoU90ZT3ImA9gYPkqbPLrGJu5lQzZ70BADxVBqJtF7yyN1W587qXpeoOnf+/JkKUZPiLjtDNvlc8Lo0JNLCdvlr9dzlNPB8tkzY9OFLHbmndWvLX/5x1yOFZqgOT8fwmftfr4oeSMzipjbF3fTcW/1uuDVCMq2rSUxA/sg9nCehKgU/kfX380UmVImIS+HqlHQBz33HsQk8uHMgZ7EZhik1tSnuZuTeEnDDpTkMcS9ky1g89+yac6tVE0mWLpKy2jJc516fFGo/IHNDST7rM2WmJsW90YzcW/xuuJ2muKeNWaFAfltGiNwJTlZxj5XwMlmX/dzZlqlbCrX8TaSkuPP3gikvNSnuDWZP99aAy4zcBVK6rhKtkTylkEBuUtUWuZda3IngcHCde72S0vNH7nI7R+5MualJcZdtf1sDHridxiFEE2kEfS51W2KNkLLFPVw2cedl9uodvYC4S1FPsLgzZaYmxV3aMkbkbngx0WQajV4XiIDhmZja19q4KbtixtqSIFrCWaqC69zrnkKeu7RlUmzLLAqEEHhwZ7/6XKuJmhT3pUEvmnwunLEsCJeWidzdTgeu2diBX7zWr6L3VEVsGa5zr3cyLX+zI3e2ZRYT+0/O4DP378SzPSOVHkoONSnujV4Xdn35ely1sSMj7sk0XA7CJ65eh/FwAj9/1Vj21RohZZdDWu+XUtxlnTtxQrVukZ97dp07V8ssLqJJOVem+j7PmhR3K26LuGsOwsVrWnFmZxC/fmMQgD1yH56JYSKcUPdD8aS6XcpJJZnGYeCEap1SqM49I+78xVgMJKXNVoV9+2te3F1O03NPpOHSHCAinNUVxLGxMADYJif9xX+9iY/+28vqfqhMkXumzp24t0gdIoRQJ/WcUkiO3Ivi6w/txUtHyraYW8nIVD9V3++89sXd4rnLHu7dbX4MTccRSaRyIqdDwyE1sSgcT6E14AaQu4LTfOA69/rGmkTNqZZJsedeDD949iie2D9c6WHMivwcq3HFrUUj7pFkWlXOrG4LAACOj0dyqhIiiTSmooYdE4qn0NHgAZBry/x61wC++9Th0xqTscyebPl7Wk/B1DDWJGq2hqtSyCqsrqgW0rpx5ROvgfcokyCvvh96zYu79NyFgCVyN8S9dzSc94zaPxkFYIh7s98Fp4Ny2g/85ysn8C9P9djaB/QMz+DoaHjWMUlbRnMQtx+oQ6xfuezvnxR37vNemFpKOkvbt5S9qUpFzYu7jNwBwOkwbq9q8wMAesciSOlCRfSSwUmjDj4US6HB44TPreVE7gNTUczEUth5YhK/c/cLGJyK4m3f3o5r/+6pWcdk7efOv+H6wxa5Z33+ynOvgai0UtSSuCcLJM6rgZpcQ9WKVbid5u0mnwutATeOjYXR6HVBcxD+57NXI5nW8bZvb8fAlBG5hxMpBDxO+N2azXMXQqgTwKfvex19E1E8vPtk0WPiOvf6xuqzZyfUU1znPivyPaoF6ypZxb2Caj9yd2YOQdoygJFU7R01PHeXw4HVbQGsbW+AW3MoWyYcT6HB64Tf7bRVy0xFk6p+tW/C2Hd5s089Lsspe4ZnbD9SXRf49a4BJFI6SNW5l+GgmapGP1VCVUal/MUoSC2Vi8oSyJpNqBLRDUR0gIh6iOiLp9jvfxORIKKtpRviqXFbbBmXI3O7uy2A3rEwUroOzYzoHQ5CZ7MXA2ZUPiNtGZeGmMVzl49bsf5Ij4yGMDITxzv+4Rk8vHtQbX+ldxyfvu91nJyOQctq+fvqsYmaiESY+WNPqBYQd/4uFCRRQ/13ErWcUCUiDcBdAG4EsBnAB4loc579GgF8BsBLpR7kqbB67prFomlv9GAikkBKF8qLB4CuJh8GJ6NIpXXEUzr8bs20ZTLiPmjaNkHLQtzWSQqHh8MYmo4hrQvbpKhjYxF1W9W5C4HhmRje973n8avX+0t01Ew1YyuFzKlzZ1tmNmrJllEJ1RqN3C8G0COEOCKESAC4H8Atefb7KoBvAsgNe8uI1XN3WWwZt+ZAImWssOS0bO9q9mFgMqqqYwJuI6FqFfeBKeMQ3rKxQ21LpgWWBb0AgMOjIUxEDFG3lmsdH7eIu1nnntYFxkIJCAH0TWQeL0QkkcL7vvs83uyfKu4NmCdCCPzBj3bg8X1DC/J69cCp69xlVFp9kV61UEsJVXkiqsZGcMWI+3IAJyz3+8xtCiK6AMBKIcRDp3oiIrqdiHYQ0Y6RkdI02rFF7pYI3eN0qFpZqxe/vNmLk9MxzJgdIX1m5L7zxCSu+daTmIokMTgZhdNB+Milq3FWVxCA8UWTydHDw2GMmxF7LJn5Ap6wiLescxcC6rWGZ+KzHs9zPWPYcWwC3/zv/XN+L06HaDKN3+4dwjOHRhfk9eoBW0I1p3FY9ZbOVQvJGorcE1WcH5h3QpWIHAC+DeBzs+0rhLhHCLFVCLG1o6Njtt2Lwm1JqDotUbzcHo6nbdF9W4MHugBOmtG5YcsY9kvvWATHxsMYnIphadCLS9a24YcfuxiA8WOU5U5HRkLKjomnMhH/CWvkblkge9qcNFWMuMvnWNXqL/YtmBcTEWNs4xZ7iZkfVkHPjuhqKSotFVPRJH7n7heKunIFaus9UpF7jdoy/QBWWu6vMLdJGgGcDeApIuoFcCmAbQuVVLXXueeKeySRskXuAXO1ptGQIbR+t2ZbrDilCwxMRtHV7DWf3/jbZFqoL9vx8QjGwvlsmai6bW0cNh2T4p7fseoZDqHXnBwlo/92c+ZsuZEnKRb30nGqyL2aE3Dl4shICC8dHceb/dNF7Z9Z0KT63yMp6tVY516MuL8CYAMRrSEiN4BbAWyTDwohpoQQ7UKIbiFEN4AXAdwshNhRlhFnYatzzyvuadsJIOA2VnEaMaNon9uJi7pb1eOJlI6T0zEsazJKH53m36Z0Y51Wn0tDShc4NGScEGTkHk2k1QkDgGWZPZGxZabzR+5v+/bTuMacHNUzbDzvQlUKTJqR+xiLe8nQi6mWqYGotFQkUnOrfsnYMqVr5lcuElVss80q7kKIFIA7ADwKYB+AB4QQe4joTiK6udwDnA1b5K5ZPXdDxLMjd3+eyP3PbjwD//bRiwAYkXg4nlKVMvKEIddpXdlqiP7eQSMKiZueu4y4G8znJ8tKTNKWGQ3Fc37s2fflSaOULYhPhUwMj4dnt4yY4pC/c81BeRbrqD9xn6uHXlN17lWcUC1qhqoQ4mEAD2dt+1KBfa+Z/7CKx55Qze+5tzdmDrPBXH9VirvPpcGpOZQNkkjpiCd19ffy+VNpgWRaYFWrHweHQqoyRtoy0ivfsqIJzx8eAyEzQ1XaMroAxsJxLGn0qvFYffqpaBInp2Pm8y6MuE9GMraMMBueMfNDnrBdGuUukF2HtkwibXyXiz2h1VJzNRmxV+OktJqfoao5SIm61aKxLuJhtWtk8lTaMtKDl2KeSOmIp3UV+WsOo+olZortihZ7olNOfpI17uevagYAzMSSKqE6Y1mrNduaOTSc8ft7hmfU7YWL3I0TTzItMBMvXdvjeiYj7o7cNVRraIJOqVC2TNGRe+3MBUioyL36xlrz4g5kRD27FBIwWgxYxT3glraMEbH6TQ9eins8lUYipau/B4yZr1JsOxo9tgodGbnvG5xGW8CNjUsbARh2h2w/ICN3IHNSkRwyBb2ryYvBqUzC1VpiWU6kLQMA4yH23UuBtGI8TkfBZfaqUQzKReI0bZlaOAFmJjFx5F4WpHViFXGPEmvdViIZ8GQnVO3iLhfN9rjsJZYyQvc4Hehsytgq0j7ZOziNzV1BZe+MhxOq/cB0NIUljcZ2WTEzHk7gd7//Ih7dM2S+nqa+/M1+l+ptAxi9Sn7+ah+SaR33bD+MH73QexrvUn5kQhXgpGqpsEbuOcvsVXGjqXIx14SqrECpCVtG58i9rEgLJl+dOwBb+4GcUkiXIe7yZCAtFHdWiaWcwep0UJa460ikdBwcmsFZXU1oazBWdpqIJNUM1ZlYEus6GgBkbJk3+6fwXM8Ydp2YBGB8kWXEEvS6bL1udvVN4vM/24Un9g/j/ldOYNvOgdN5m/IyEUmokyKXQ5YGWS3j0hx5JjHVjuVQKuZsy9TQalUJjtzLS77I3S7u9ojeQUaJpFtzqAobub+0UDym6MvHZCTt1BzobMp0iIwndRwankEyLbC5K4i2gD1yN2yZFNobPWj2u9REJqtVAxhfZPnlD/qcNnGXVxMnxiMYmIyq+6VgIpLEarP/PVfMlAYZubudds9dCFGXa6jO1WaR++kit5qs2khV8ee5OMTdXCTbGqG7C7QCJiLlu0tLBshE6jJy92RF/lJsXRphmRm5e5wOxFNp7BkwyiLP6gqixe8CAFy8ptX03I0Zqo1eJ1oDboybHre0Q/7ipjPx1jOWGOJuRnVG5J75ssirhjf7pxBL6rYE7XyZjCTUVcV8bJkfv9CL53u4hQGQafnrzkqoWqO7erRliu2EabU4qt2aqeZSyMUh7nlsGVntYn1cIq0Zf7HirpFKqDodDnSZ4t7Z5EU8pWPvwDT8bg3dbQE4NQe2f+FafOfW8406Z92olgl6XWj2uTBlirpcx/Ujl69Gd3sAybTIRO5eu+cuX/uV3glzjPaofz5MhBPobPLC59LmlVD9x8cP4YEdJ2bfsQ6QCVVXVuRuje6qMdIrF3OtELKe+Ko9qaquxKrwCmNRiLvy3IuI3AHAbyZVrZG7w0Fwaw4lnLZqGc1qyxC2drdi/ZIGnNkZRCypo38yilWtfvU6q9r88Lk1dDX7EEmkkUjrCPqcaPa7MRk1BHQinIDfrcHj1ODSHEicwpaRr21d+7UUa7Om0jqmYyk0+93GVcVpRu4yabxQFT7VTkpF7vZJTNJLBuozci82Ck/UYORes4t1VDsyMrdNYtLskbcVactYI3fAOCGEVOSeeczpICWwLs2BMzuD+J/PXo2lQS/iqTRCsRQavbnzwS5c3aJuN5qRu7RjJqNJNPtc5ljJtGXS0BwEv9tpi9wjWTXvusjddjrIq4cWvwttDe7TtmXiKR2JtG4bcz2jW6plrL95KVpOB9Vn5F60LZP/aqcaUb1lqvBkvUjEXXrusydUgUw5pN9lF2S301HAlnEgZqmWkXhdGuIpHaF4SrUdsLK5M6ieJ+h1osnvUoI6GUmgye82x++AEEA0ocOtOeBza6qtAQBEE7keeyieQjiewiV/8z/YftBonzweTuDBnfYFQWLJdMGGZXICU0vAbVxVWGre9wxMFf1jlO0VFmriVbVjS6haO0SaQuB3a1UvWqVk7r1lase+qubFVxaJuOfz3PP3nAEskbsnK3LXHJZqGastY4/cra+RMMU9kEfc3U4HzlneBAAI+lxo9rkxE0shldYxGUmq5KvbMuHKpRG8Tg2JtK5EIl9EPBNLYjQUx9B0HC8cGQMA3PiP2/GZ+3eqEwgA/NMTh3Dd3z2N4elcgZe1/m0BD1r8LiX2o6E43vVPz+Kh3cWVXMr3jCN3A2spZFoXykKTtozf7axKMSgXc20nkNRryZbhUsiyIsXR5rkXaAUMZJqH5bNlMnXu+W0Z2wnEPAGMheJ5bRkgY80EvU40m2I+HUsZtox5X54wQokU3E4NXvN5pe9utWDkoczEMh73kZEQZmJJDJk19HGLyO7onUAonsLfP3YwZ2zHx402w6ta/WixRO6joTh0AUyEi0vcTkWN96waxf3rD+3Fjt7xBX1NqdvyOyh/9zJyNSJ3UZK8SS0w58g9VTsJVa6WKTP5IneHgyx2jf0wZfMwX5Yt43E6CsxQdSghddpaHBjPMx3Lb8sAwDWblsDjdGBli1+J+WQkYdgyPtOWsUTubo1UoveFw2N4bM9JRBNplU/obgsAMMRdiunhkbBtfVbZEkEIgb2D0/A4HXjg1RMYnIriM/e/jh8+dxSA0Q/H6SB0NXvR5HNh2nJVAWT66aTSulrcJB8qcq8yWyaWTONfnzmK/37z5IK+rrRiZNAhr8BkBCuvGKsx2isHqhTytGyZ6n6PVDsJTqiWh3yeO5B/5iqQaR6WL3KX2KtlKO9t6z4NHlfesV22rg17/vodWBI0BBQwvG6bLWM+ZySehtvpgNc8afz9bw/iaw/tQzSZRmeTF0uDHly2rg2A4blnmpaF8SvLrNVMp8ooZmIpvGVDO4QwbJjnekbVknrHxiNY0eKDU3OosUxFk8rWkb7/f+44gav+9klVrZON9NxjVRa5T2aVnS4U1jp3IGPTKHE3v3/1Ys3MOaFaQ7ZMkhOq5cVtimG2ty5nmWaXQsoFO04t7vnr5J1a/hNAQwFbxvo3UtwHJqNI6SLXlomn4NIc8Jrj6huPYDqWRCSRRoPHiWf+5Dp84up1xr6WyD2ZFnj12ATWdRhRvex3s2fAWGT7AtMaiiV1xJK6alB2fCyCVeaVQEsg0zZhKityf/34JBJpHf/1Wl/e41MJ1Sxxf6NvcsGF1YpsirbQY5ARuZxcl8qO3M3Pt9qj0lIx51LIVA1Vy3BCtbzMFrm7Cnjuvmxxtwh3od402dUykoas5Gw+ms3qGLmkXrMvUy0DAOFEyozczQlV8RSmo0lEE2n43BrcTgeCPunbJ1UFj+Ttm5cByPyI9gxMQ3MQzl1htCGOJdOIJdMYmo5BCIHesTBWm2u1yhPPVDSRE7nvP2nMwP3Fa/15feLpWMZzl48nUjre970XcM/2w7O+L+VCint2q4dyYy2FBDK2jBStQL0IG9reAAAgAElEQVRG7kWezKzvS9VH7pxQLS/uPHXuQEagtSzPPd8MVcDeT6awLVMgci9gy1iRde29Zu/37Mg9bNoy1pOOLowEp88cm/T2Q/GUiqwB42rk0rXGcoGJlI6paBJPHRzG+o4GJdzheAopXWAsnMDITBwzsZTqK9NinngmwhZbJpVGKq3j4FAInU1eHB0NY1ffVM5xychdiIwldHIqhkRKVytLWXlwZz/u/PVePLYn1wuPp9IYC9l73EQT6dOKvjO2zML2qc/23PWCkXt1C1epyFTLFGfb2WyZKn+PkpxQLS9SHF1afnHPncQkZ6hm1bkXEG6nzZaxVstYIvdT2DISGXUfGzMjd1NQ3eble1jaMi77SWdoOqYEQXMQAm7NSKgmdHWcV6xvV8IfT+n4nbtfwP7BGXz8LWtU9Y1VIGUrg1WtWeIeSahZtPGkjt6xiBGFX7gCAHB4OFesrZGx9N3lSve95rFa+epv9uHe547irx58M+ex7z51GO/8zrO2bXf+Zi8+8oOXcvadDRW5L7Atk87y3NM5nrsp7qnqE4RyMNdqmURKqPeu2k+AmZWYqm+ci0rcsyN0jyqRLJBQzRJRjzNzBWAVdKut48qzIAiAgtUyVjQHIeh1KsHLjtyjyTQ8ToeK0iUTkaTtRNTgdSIUyyRUf/ixi3DnLWfb6uX3n5zBJ69Zh/dvXanyB1Zxf/moURvf3W547k22hKoR6cZSaWXJXLOpAwDUMoBWpi2RcVSJu5F87R2L2Bas0HVhLmQCDE3HEc7qcLnzxCROmraRZN/gNPafnJlz6WDFEqoivy0jhUp+ltUoCOVg7isx6aqiqPptGeOzFQI5C7NUmsUh7s4CnruK3LNLIY0fVyB7EpO5vztrf2u0XmiiVKE692ya/W61ClS2uMvbXlfux+JzWV/LhZl4UgnpBatasKzJaynNNMQs6DWeX14JTFpE7skDI3BppCL3oNcJzUFG5B7JRO77B2egOQhnL29Ck8+VtyTSGrnLcsg+s7ImkdIxMBW17ZvWBS5cZSR5syN7aeNY+9T0TUQRT+kYCc2tJfGE2U4hFE/ZOg1Ox5JlrTG3zlC13peec6DObJnMLM7i3vOUrtdMXiKp68oOrraT9aIQ90Ilj5mGYvbty1t8cFDueqhyf48rW9zt4iuxVtTkm6Gaj6DP2G9Vqx/tZu9363O6NYfteSV+a+TucZqTmNIgypxkpJjISFWeJOT/1lWXjo9HcM7yJiX8RIRmnzFLVZU2ptI4OhbGqlY/PE4Ny4LeApG7RdyzbBkAODqaEXDZnExW8PSOZvYLx1Oq3DJitlyIJdNqYRV5NQAAn//ZLvzZL3fbxtE/GVWVPkCmvQKQ6fY5FUnikq8/jkfKWPueY8tkrdYjE/r1Y8ukzf+Ln8QkratqjtyN2cdQV9rV5rsvCnHPLNZhP5xCnvua9gB2fvl6nG22BpBIUbdG5ECWLaNZq2XmZssAwJv9hs3xx2/fAIf5vNbXy06oSqzbGr1OVefuc2kgsj+PtElkTkAK+FTU3hjs4jVttvtNfqMlsbVaJhzPNEVb2uTFUD5xj6VUnXzMYsvIZG1vPnE3FxK3Ru5HRjK3szthyueU7Ogdx2vHJtT9VFrHu+96Dt98dL/aZu2VI4+pbzKCaDKdN9FbKtQyezKhWshzr7JIr1zMtc49qevqBFhshU0lyNhsNSzuRHQDER0goh4i+mKexz9LRHuJ6A0iepyIVpd+qIXJtxITkImss7cDGcvCiorcsyJne0LVErnbSiGLE/ePXt6NRq8TN5+7PGf88nZ2QhWAzYdv9DrVDFVv1opRQMYmkY+5NAc0B9kidwC4ZE2r7X6L320mVDOReySeVmK0LOjJb8tEk1gaNHrcyyRv/0QU569sht+t4aglOpfivqLFjyWNHltULxcLN57H7t0bt43nEULg5HQMQ5aGaC8fHcfITBwnxjOvZV38W74nsp/OSKjwjNv5oqplNHuduxSqTEK1PsRdXqEk0npRdlgyrdeEdSXHJoO8ajtZzyruRKQBuAvAjQA2A/ggEW3O2u11AFuFEFsA/BzA35Z6oKdiWZMHjR5nTsSbSagWd4HizrI3JDbPPc8i3H63llOGWYiv3HwWdn7petv+1qsBa517o+WEYS3bbPDIhKpuE31Pti1jOQ6v06HE3ekgEAEXdmdaEgNG69/xcELZLPGkjkgypfzPZU0+jITith+cEAJT0SSWSHFPGuWTJ6djWNnqx+q2gC06l4LbEnBjTXvAFtUfslTiZNs7moPQbwq97B0/GUkikkjhyQPD2LbLmKE7allwZDKSVAuTy/dEivvojLFfIqXjwMnMSaUUZNe5q1JIU8wzfnJ1RXrlwlolU8wxG7aMGblX8QlQRuryN1htSwIWo3oXA+gRQhwRQiQA3A/gFusOQognhRAyZHoRwIrSDvPU3Hzucjz7p9flRLyFbJlCyP2zbRl3Qc/duF1s1C7JPhHYPXejUselEZY2eZWo220ZF2ZiRkLVmh9QkXvUHrnL21LgLl/fjus2Lcm5emnyudE3EVWNrmTk7lORu1e1MZBEk2mkdIGlpohGk2kcHgkjrQusaPGhxe+yrRwle8a3+g1xl5H7dCyJF83ulkCmWVrfRBROB2HT0kYVxVt9/39//hg+9m+v4P5XjFWgrGObiCRULx4l7qG47f/7Xj6Od37nGZV8lewdmMbl33j8lD11CqEah8mEarYt4ylNVBpLpquuQiMfVoEuphwyqevwuoy1jmshcvdVafK3GHFfDsC6flqfua0QtwF4ZD6Dmiuag1Qpn5VCk5sKIe2YbHGXkT+R/bnk/nMV95xxOnMF2uvS0N7gVgJsjdCbfC6EE2nMxFK27fJ4p/KIu8fpUNv//KYz8IOPXpQzjo5Gj2qc5tLI8NwT1sjdEHAprlORJD73wC4AwPolxjqsLxwexbvveg4ujXDBqhb4XJqtLcFEOAGfS4PPrWFNewBj4QTGwwnc/E/PYueJSbzznE4AFs99IoquZh9Wt/lVFG8V92d7RkAEbFraiIvXtGI8HEdaF9B144pC+v7Zkbv8f3f/FFK6wHGLnWNsn8TAVMx2wikWKebye5PTOKwEYiCEwDXfegr3mk3gqplEOnOFWYwVlUwb6xrIFcqqFbm0nqxkq0nPvViI6EMAtgL4VoHHbyeiHUS0Y2RkpJQvnRcZ1c7Vlsn13A1Bd2U9j0sz7I1iJjCdClvkbo7B59LQ1uBR1TVWW0aWUA5Px2wCTkRwOx2qHYA14et1aUq4vXmqcQDgf23pVLeXNHoRS6YRSaRVpCl99aGpGIamY3jf957H/+wbwp/csAnvvcC4WHt490nEUmk89sdXY8PSRnjdmq1b5Fg4gVazj43sdf/AjhPoHYvga+8+G59+63oAVs89guXNPqxo8aFvIgohBIYs0fSO3gmsaPHh0T++Cu/a0gldGL7+dCwJXUCJu0wyK1vGjNwPDRmWTHZTNGnv7Dwxmfe9OhW6LuCwBALZpZBK6OYhBpORJE5Ox/D68bmPbyERwlgbOKASpEWIe0rAqRnf5eq2ZczIXVbL1JrnDqAfwErL/RXmNhtE9DYAfwHgZiFE3oJkIcQ9QoitQoitHR0dpzPeOVGoFLLg/gU8d9W7JsveISJ4nI55R+752ht84R2b8PtXdGcid0sppGwncHI6ljPhyaM58kfuWRZNPqzVQ+2NHsRSuiHuFlsGAAanYrj3uaM4OhrGv//+xfj/rlmv3oOpaBIdDR6sMSdH+VyarWZ9wiLu565shoOAHz7XCwB4y/oOdTyRhOHd9wyH0N3uR1ezD/GUjvFwwha5x1M61rQbVw3tDcaVxchMXOUXOpt8cGmUE7lHEmmE4inl8/dP2MV9zBT3N/rs4tk/GcWHvv+SLXGbTVoIOB0O9b2zRu4uU7TkfeO14nNeXFzaSodHylf1UwrkCUxWXBUj1ildh0tzwK05qs7qsJJ7JVbcyfpzD+xSOaJyUoy4vwJgAxGtISI3gFsBbLPuQETnA7gbhrAPl36Yp8dcPXePqpbJb8vkO0l4XVoJxD03cn//1pW4cHWrEvJsWwYworfsCU8elyPjuTutgp5/Zm02X3mXkStf1x5AImWsBiW/vK0BN5wOwkgojvFQAh2NHly+rj3nOTubvOq2tGVC8RTuffYohqbjStwDHic2LQvi5HQMHY0erGz1KX8/mkzjteOTmI6lcOX6DrSZwj0RMcS9NeBWx7TWPJF0mL7/SCiOZ3tG1ViafC6b525WjuKNE5PK28+O3MfChni+OTCNbbsGsG/QKGG989d78KylbbKVB3f2YyqSNCJ3B1SpqyyFHA8ZPfzlyVyKw09fOo4/+fkb6mqiGIbNhVmOjoar2neXxygnDBYTuSdSprhXeeQuxdwzB1smlkzjF6/12QoJysWs4i6ESAG4A8CjAPYBeEAIsYeI7iSim83dvgWgAcDPiGgnEW0r8HQLSr4VmorZ35MV2cofo1WEJT6XhsY8ZZVzIXsSkxXZj8Zqy0hxB3KjcLfmUM27vG5rtczskTsAfPSKNdj15euxzvTQgcyMSiJC0OfCdDSJ6VjSlpB1OEiJ7TKruJu2zAOvnMCdv9mLvYPTStyBTL37hataQETqJBZNpPDE/mE4HYS3bGxHq9n7ZiyUwNBUDEuDXmUTre2wi/uuE5P4xsP7cNnaNly6tk2NGTAid3lV8dxhQ6AdBJwYj+AD33sB9z57VL0OYAjNH933Or784B68cHgMj+4ZAmAv2wSAnuEQPnP/Ttz1VA9SuoBGBI1k5G7s0zsWxuo2v/qMpdDJq4fsVgynQq6LG0/pBfvsny7bdg3g+FjhK5O5IMVZBkDFRe4CLo3g0hxVXVGUXS1TjC0jP6uVrb7yDcykKNUTQjwshNgohFgnhPi6ue1LQoht5u23CSGWCiHOM//dfOpnXBhUnXuxkXuBahlZ257veb7+nrNx+1Vr5zNMaA5S/my2JRT05rYnlg3HgFyhLmS/FBu5A8bJw/q3/ixLaDqWwnQ0pfIBEvkll/aNHEN2Pb5d3I1yTLkcYUbcdTx1YBhbu1sQ9LrU38jIfVnQg6WNxutIsZa2zN1PH0Y8peNb798Ch4PQFnDj9eMTODg0g5lYCps7gwCA53rG1BheODKGl3vH8bWH9uKZQyMYDcVxprkfYLz/0qJZ2xHImQQlH/uv1/uRTOtwWD5T+aM/Ph7B6la/OmFL6+ig6fuHLOI+HUuqfvz5GLZUBZXSmgnFU/ij+17Hj17oLcnzych9LuJu2FdGxVg1J1StyyYCxbX9lXZe9uz4crAoZqgWIhO5z89zl3+f7wrgujOWYtOyxvkME0Dm6qBQ5J7Plsnenv33+aJ1t9Oh7IJTYT0BWBcSD3qdmIomMRNP5pRSKnFv8uVsm7TMjrWK+9WbOnDp2la84yyjF73T9Fr7JyPYf3IGV29cYvub8XASQ9MxLGvyYWmTjNyNq4yAxwm/W0M4kcYFq1rUD+gL7zgDkWQa1//f7QCgRHvniUksafRgc1dQ2TPLgl585/FDGAsncN7KJvz8E5fhrK4gkmljti4RcN6K5pzI/Q2zFfLITBzbD47YTti6blyOn5yOYVWbH16XhkaPEyMzcaTSOo6Yl+jheGZZw4/e+zJu/ufnVMIXAJ4+OIL7Xj4OwLBl5Mdondk7X2TNv/XkMR/kVWSxCVUhBJJpYXwPnFpV2zLZCdVi8gOynHdFS5VE7rVKocZhs+2f036gQEvhUqIWFskaa2vAnVORE7Tczvbc5TE4KLtVgtmKYJaoPXt/IDPpBkDGlommcpqlyRWk7J678XrS5tAcpDxywIi277/9Mqxqy0QyPremet6rfvOBTBJ5NJTAsqAXa9r8aPK50Gm5UpDR+1Ub29W2i9e04qcfv1TdP8NyMn7/1hVY3mz80NZ1BHDtGUtw4OQMxsMJtAU82NrdirYGD8LxFMKJNPwuDRuXNWJoOm7rNvlG3yTOXdmMJp8LvWMRw5Yx3+q0EOibiEBYqnc6Gj0YCcVxYiKqBCxs9tO5e/sRvHZ8EhoRvvbQPgBGBc5f/mo3vv7QPui6wPCMMUks6HXOK3Ifmo7Z5iHILqAjcxD33tFwQQ85kRW5z1YKmVJ9eQhujao6oSrH6p1D+4G+iagxh6XRO+u+82VRi7tnrtUyBdsPyGqZ8r1dha4a3nfhCvzktktsUbJTc6jZqznVMpY6edlzxrif2V4M1hOc1RKyee6+/JH70qA352/HQnG4NMLOL70dN5y97JSv7XdrGDC9Sdln3uM0Etd7TatieYsPn7hmHX7z6SttVyLSd3/LBns11uauoCr1tF4Sf+ra9er+FevbsWFJA6ZjKaR1gbYG47UbPMbVQDieQsDjxAYzH/HBe17EX/96D1JpHXsGprF1dQsuN9e4dTgIDuW56zimTlamhdTowehMXFkyQMZz//WuAVy2tg2ff8dGPH1wBPtPTuO5w6M4MR5FKJ7C0bEwRmbiWNLowfolDeqqIRRP4epvPYntB+1lxkII3PHT1/D4viHb9vFwApf8zeNqrgIA7B80xpOvA+fTB0dUYtnK53+2C5/66Ws524FcWyY+i1jL/UuVUP3PV47jr3+9Z17PUYjEaZRCnjBLe4u5ep4vi1vcXXObxDRb5F7sSeJ0cBWI3Bu9Llyxvj1nfzlpKzv5a50EZUWesIoV90KRu6w8mYmlCtoy1shdPs9YOAG/24lGr8t20smHz6WpmaFWC6cl4MLufkPIupq98LudWNlq9y6XBb1o9rtymsIBwD/8znn4+Scuw6Zljbj7wxdi2x1XwO92YuNSQ6yvPWMJ1i/JRPWyQsfvdiISTyEUT6HB48QGc5+9g9N4ePcgDg2HEE/p2LKiSfXrGZmJo8u8Ith/ciYj7uZ4OxqMyL3H0nIhEk+r5Q83dwXVCerQUAj3vXxcXYnt7psyxd2Ld27pwu7+KewZmMKb/VM4NhbBjmMTePnoOP75iUM4ORXDWDiB37wxiCf22wvZvvGwcVWwZyAj2NKWyRe5f+Fnu/CZ+1/P6Q9zaDiEPQPTeZvKJbJtmVnEWvahcWoOtDfk72U0Fx7efRI/35FZ+/flo+M5K32dLjkJ1SIj9+zvbLlY1OJ+3spmvGVDu+pZPhtqhqorW9wLV8uUCleBMsxC5CuRtP59tv2ibJk8veLzUdhzd2EsnEBaF7kJVVkPn1UKCRhiEXAXd2LxuTV1ySvtGMBoWTBklgBKKyWbz12/ET/4vYvyntCdmgNbuw3xfcdZy7DFXFt2w9JGPPun1+LaTUuwYWmmSqhdlmy6jQlgkUQaAY8TK1p8WBb0YnmzD0PTcfy32T74vJXNuGRtptPm0qAXZy8P4vF9wzg+HkGDx6lOVh2NHozMxHFoaEZdhYXiKQxNxxFL6uhu86vWCUdGwth+cBTvPX8FvC4H3uibwvBMHB2NHrzvAmPbT148rqLqwckovvf0YfzdYwdxwz9uV8leq48+MBnFz17tU2MBjAh/38lpEBnzFeKpNPomIrjth69gNBTH8EwcB4dCeOpA5spgIpxZc9d6xfCv24/gif1DOdUys9kssvmWWyOs62jAsfHIvKL3k1MxzMRTmIomkUjp+ND3X8I/PdFz2s9nRXnuc0io9o1HFsRvBxa5uK9uC+DHt11SdK/1got1yDr3MnruKqFapLjLWaqF+ulkb5+rLWOvlsmfzM0uAfW6NDT77ZU2ypYJJ4r+HKwnrBZLZZA1ireeQKys7WhQlTdzQVozSxo9SmzbTdELeJxq0pPfrcHhIDzzp9fi7z9wLgDgxy8eQ1eTF6ta/di01J5cf+sZS/Ha8Qm8eGQMq1r96qqlo9GDmVgKu/unsGWlcZURjqdUk7Xu9gB8bg1dTV48c2gEoXgK561qxubOIF7uHUMonkJHowdNfhfetaULD+7sV7NVB6aiODYWRtDrxGQkiQd3GhNmhqftXTQBI88gxbl/MoqZWErNHB4NJfDSkXE8vn/Y1v/+B89mWh4ctTSFe9oU9x294/j6w/vwzUcO5HjuiZSO+18+jhv+YTt29I7nfA5WW2bdkgDSusDx8QJ+fkrH3U8fVjmDbbsGcq4s5EIx/RNRHB+PIJHWT2vWcd7Xn2NCNZJIYSycWJBKGWCRi/tc8RSocy/UfqCUFLJlCqEid7d9/8zVR7a4y4Tq3D13vy2hak3s2sX9ou4WvO3MpbZt8os/Hk6oHt2zIU8IjV6n7f1osUS9+RY0KQVEpGr82yyTrVK6wEQ4oUTKpTlwVpdRdTMeTuCyde0gohwv9W1nLoUQhjXz/q2Zfnrtpp9/eCSMzZ1BuDRCOJFW6+vKqH1NRwA7zL71Z3YGsWVFs1oTQHa8vPm8LkQSaTy8exCAIWQnJqK43qxA+u1ew2uXVz0A8HLvOBo9Tly6tk116vz1LuPvZX+fkZm4mlgle+ysbvPbfHc53nNXNuO5nlEIIfDV3+wFABwYmsFe0/KRJ/Z4Sse/PHUY+0/O4NZ7XkTfRAQDk1F1gpHWhlNzYJ1ZBdUznF/cn9g/hG88sh+/fK0fR0ZC+NOfv4EHdw4oqysUT6lFWvomIirpu29w2rYy1+mibJkiE6rHVRkkR+4LTkejBxd1t+DcFXa/Vgmvs4zVMgUSqoVo8hnikFMKqSL3LFtGnbjmXi1TKHLPtmVuv2od/u795+Z9nrQuirdlzL+xRupARmy7ClgypWLDkgY4KDOfQB7/8EzcdvXR6HWpyh+ZSAWA1/7q7XjmT64FAJy9PIh3n9eFr95yFj52xRq1j7RCjNdrRMDjNCP3CFwaqbyFrOF3mM3Rrj9rKZYGPWj2u3CO+T29dG0bgl6nsgXkoubnrWxGV5NXlXmOhOJ4cv8wPvvATrx4ZAwXrG5BW8CtLIufvHgMl61tU7OOh6djStxfMsV9y4pmjEcSShyPjkbgIODaTR2YiCRxeCSMXX1T+MOr14II+NVOo1OJrPbafnAEx8cj+NgV3UjpAnsGpvGh77+Ez//MSOomVOROStwLVQPJK4VnDo3gL3/1pgrCnjxg5BZOWpZ37J+Mqg6k8ZSuJo5NRZLY+rXfqhNjPvomIsp6syITqNaEat9EpOD8hNeOGVcM5+TJB5UDFncLXpeGn33icuXFSk5V514q5lpuKUW2YEI1K7LNXpVpNjwWi8oaPVuj9XwLnmRjrbSxXgGcCimmVksGyETuy5vLW0b24ctW489vOlP59gFL35zsdXfPMn+ol1nEvTXgVkkzIsI/3Ho+PnxZt+3vOhoyx7B+aQMCbifCiRSOjYWxssWvKrNkBC9tmsvXteOlP38bdn7pepyxzLhycGkOvNW8YtpiCUxWtfptieW0LvCvzxwxI90wLl7Tima/G0IYlkb/ZBS/d3m3rY2DLGGVjdS2LG+CEEZJ6jce2YcdvePoavapE+5eM6q/uLsVF3e3qquMBvN9e+TNk2j0OvHJq9cBMKLoI6NhPL5vCIdHQnjSTPq6NAcCHic6m7x5xV0IgadN73/7wVE8f3gMn7p2PTYubcAT+4fxXM+obQnH/okojoyG1We626ww2n5oBKOhBH75Wl/Oa0j+7tED+NRPX8uJ9uVJU/6mvvN4D6762yfxnruety33KHnxyBiWNGb6LpUbFvciWIg6d/ncxSZUpedeuBQyO6F6ep67P0vM7J777GJtHV+2MBb8G3f+yF22IOhqKm/kvmVFMz7+lsysY2u1UCDrBPW7l6zCJ65eN+erifbGzLFtWNKAgEczIvfRiKqFBzKtFayzZfPx7vOXw+kgW2dPq7jLiqAdvRNq8tMla1rV9+iZQ0br5GvP6EBbgzG3YmQmbiuJbAu4sdy0FB7bM4S7nz6C5w+PYU17AB1mZdF+U9w7Gj14z/mZzuANnsz35n0XrsCSoBctfheeNAVaF8C7/ulZVdffbH7P1nU04PBIGLFkGt94ZB+OjRm3H90zhIGpGK47YwkSaR1upwMf2LoS125aghePjON3v/8S/sasBvK7NfRPRtE7GsY5y5vQ4HGqqisZ/T/bM4o/+NEOfPTfXoauC/zi1T61sPpTB0eQ1gXG5cLxKaOP/q92DmB1m1+V/vZPGpUwibSOx/aexIM7+9U6AUIIvHhkDJesbZu1WqxUzK/jVZ2g6twXIHJ3a8UJoBTZohOqshSyyJOHWmUq63mCNlumiMi9QBuDU/+NsV925N66QLZMNtYTXHZS+FKzf81caTMXR+9s8qLR6zJtGcNzv9iy/OFas+Pl5lnE/eqNHdj55euVr+wg4306e7nxd5etbcPBoRASaR23XbkGV23swIWrW5QnvXdgGh0NmVxGq99teu6ZmcWdzV41SczaLbO7LaC277OI+zu3dOKL5iLmVmvvtisNe6q7PaCSwMuCXoyE4vj2B87Fuo4GZV2sX9KA+14+jk/+5FU8eWAEQa8LfRMR3PfyCTgI+LMbz8CzPaN415YutAbcePf5y/HbfUOYiaXUzN9zVzSjfzKK4ek4rljfDpdG2H9y2oj+D46gs8mLwamYyk3s7p/C5362C9/+7UH8/QfOVW0iZPnp+7/3AsZCCfRPRvG1d59ts1JvvWgVfvLiMfz/j+zHWDiBP7puPT57/Sb0jkUwPBPHpWvtS1uWE47ci2AhqmWkmBbr669pD8DpICwNemzbC9Wze+doy2Qid7uYzTlyt/yoi/Xc/Spyt588pF2wUHXCEmvXz2KvPmbD7XSgxe9Si5wE3E4MTkURTqRtCbfVbX58473n4IMXrypqnNKr72r2we104Ir17fij69bjdy/NLGt89vIgrt7YAaLMIjeHR0K2k6Ys1RwLxZWV0dXkU4lgOXHqz286Ax++bLW6EtlnToJqC3jQ6HVZmvEZ/zf7XapaRFpORMB//uGlePBTV+C9F6wwWkGbr3nblWuwstWvIvwT4xHsG5zBOdoow6QAAA0JSURBVMub8PBn3oINSxvxy09eji/fbHQzPbMziCc+dw3ee4Fx1dDe4EF3u9EL6OR0DGs7AljR4sfAZAz7BmcwMhPHp6/bYFvSctCsre+fjOKTP3lVbR8NJTA4FcUbfVMYmIqivcGD9124wnZFf87yJtxw9jK14th2s3voQ28YFUuXrJl7IHC6cOReBAtZ555dhlmIS9e24dW/fHvOClSFbBmPK//2QsjnyRZkKehel6OoihWrzVR0KaT03LNsmfNWNuN7H7oA124q/1oAVqxRZ7HHUAy3X7UO60zbJeDR1EQna5knERUl7JLWgBsep0PN7fA4NXz2+k1IpnUQAUIAm5ZmrgLk1ZEu7HMHupp96B0LYyycwJmdjXizfxpdzT5VHnpkNIz2Bg9uv8rwzmUt+snpGFr8GVF//LNX4xev9aGjwYNtd1xh85ul/bSyxa9m7mazstWPBz91BV48MobvPNGD4+MRHBsL48ZzOlXOId+Etbes78DdTx9BV7MXK1p8amWvdR0BhOMpnJyOYXe/cdVw2bo23P3hC/HQ7kH8x0vHcWTU8Pg/ec06/OLVPmzuDGLv4LQ62QHATz9+KdZ2BOB12fvfnNUVxJKgBzt6x7GmPYBtuwawo3cc33miB9dvXqpO5gsBi3sRLOgM1SJtEwB5lxbMdLacX+TucBDcmiNn0XGX5kDArRVd1ijb+EaT6eI9d1fGGsh+rhvO7sz3J2XF6rPPt3e/lU9es872GrLaxdpVc64QEa5c347zV9mLAlyaA20BDyYiCaxbkhHSZsuVWJclUX1mZ6Oa0bp1dasp7l40epzwOI220tbEttvpULOXrZVAK1v9+D9v2wgAOYUKMnKXeYVCBDxOvPXMpfj1rgE8fXAEE5GkmulbiK3dLfA4HVgW9OJ/benEwGQUZ3U14bozlmIkZEzCe/WYkX9Y3uxTSz7+x0vH0WN2/Pz0devxhes3IZxI4ZyvPIbRUByHh0No8rlwyZpWdXVhjdxbAm60BNx48I4r8eqxcfxq5wA+cu/LaPA48fX3nHPKMZcaFvciWIjeMnON3AuhlulzF/DcixR3wIj2sxOIgOG1+4u0WORYosl08Z57gci9Ulij9WKPYT6vUWiCVrHkWx8XAJYGPWjxu2wn/qDPpSJ6qy2zuTMTDV+4ugWr2/x455ZOEBHaGzzon4yq5KqkvcGdI+6nQkbusuRxNla2+jFh+t+FIn2J16Xhb95zDla1GVcFVmHtMt/fl46OK/sKyOR0Dg2H4HNp6rNu9Lrgc2kYmYnjhSNjNmEHUDBBeu6KZjR6ndB1ge//3tai35dSweJeBHLyUlm7Qjrzt/yd8/No+UshpR1TbDWOsW/+CL3J58o5eZwKGYkXG7n7C1TLVIqALaFanslT1qTtkjJ1DPzE1euQrUOag9Dkc2EykrSLe1fGumlv8OBd53Zl7jca4p5dtdTe4MHhkbBKrs7GuiUNaPQ6cVF3cTOKrbkWa0VRIf73hSvybpfHeWwsYpufIL9vPcOhnO9eR6MHu/un0DcRxe9b5itYx3PrRXb7zKk5cO9HL0LQ6ypJW/C5wuJeBAtVLeN05M5wnCuFvHVpJ8zFVrhyfRsuyDOV/8LVLXO6ApBjKTbqPXt5E85d0aS6L1Yan0tT0W0pbRkrDeZ7097gKXoi21yxCrSVZlPcrZ776lY//G4NkURaJVElHQ35q5akH99RpLgHvS68/ldvL7qx36o5inshrCcl63NKQY8mc4+5vcGNV82ZwueutNtLAPD0F67N+1oXdS9cdUw2LO5FsBB17j63NieroxCylDJbfJcEvfjehy7ElRtyO0wW4h9uPT/v9rl6hzLKz2fx5GNdRwMevOPKOb1GOSEiBNxOs7dMeX4y8gppWdPCXroD5kzcsYhNrB0OwpmdQbx6bCInEpf3s8Vdivpc7Ie5WJ1SiDsaPfP6HII+pzpxWdcRKNTHSL5mWhfQHDRrWWq1wKWQRaBmqJZR3H//ijW4+8Nb5/08haplAOCGs5eVLfI8Fb4CE6JqCXniLVvkbr43y4ILW8MPwGz2ZpRmWjlneRM8ZqLUihT37M6cMtotl7e8NOiFW3Ogex5RO2CcrGXJqDVydzsdqhqsLeuEJo9pw5KGOVmSlYQj9yLQHIRzVzSp0qtyYF3weT4UmsRUSeRYio3cq5GAxwnMxMvmuQcqGLlvWtqISDydkxi847r1eOeWzhyrsLs9YCu3lLSfRuQ+FzQH4dyVTWrd3fnQ1ezD4ZFwzjG0BtyYiaVUHyOJPLaF6gtTCmr317aAEFFV2QSnolApZCWZa0K1GpFjL1u1jPm8nWVurZCPL954BvK1Im9v8ORNjr7n/OW4Yn1bTinu5q4gPE5H0dUvp8MDf3hZSZ5HRu6rW+1VN60BN46NRfLaMoC9d0+1w+K+yOhq9sGlUdELlCwEc/XcqxG/2wmfSys6+TdXZOReiqu3uUJEmIvjqDko70loy4pm7P/qDWXtnVKq57564xIMz8RzTlAyYs+2ZWRN/oWrK5cgnStFee5EdAMRHSCiHiL6Yp7HPUT0n+bjLxFRd6kHyhTHylY/9t15g62UrdIsBs894NZKOjs1mw1LjEVGLq5gdUUpWKimWPPlnVs68cOPXZyzXSZVs22Zy9e14anPX1NVv6vZmFXciUgDcBeAGwFsBvBBItqctdttACaEEOsB/F8A3yz1QJniKedkq9PB69LgNGe81iqGRVG+uvuWgBu/+OTltuoNZuFpbZCRe+7s6O4FatVbKooJRS4G0COEOAIARHQ/gFsA7LXscwuAr5i3fw7gn4mIRPZKukxdctM5nQh6nTUT1eXjT288A5F4utLDYMqMbHlRLRPo5kMx4r4cwAnL/T4AlxTaRwiRIqIpAG0ARksxSKa2uXhNq62NbS3S3uABqmNOFVNGbjqnE+FEuuAC7LXEgl4nE9HtRLSDiHaMjIzM/gcMwzALyMpWPz779o01fZUpKUbc+wGstNxfYW7Luw8ROQE0ARjLfiIhxD1CiK1CiK0dHQvbtpVhGKaeKEbcXwGwgYjWEJEbwK0AtmXtsw3A75m33wfgCfbbGYZhKsesnrvpod8B4FEAGoB7hRB7iOhOADuEENsA/ADAj4moB8A4jBMAwzAMUyGKKtwVQjwM4OGsbV+y3I4BeH9ph8YwDMOcLrVbeMwwDMMUhMWdYRhmEcLizjAMswhhcWcYhlmEUKUqFoloBMCx0/zzdtTn7Nd6PG4+5vqAj7l4VgshZp0oVDFxnw9EtEMIMf9li2qMejxuPub6gI+59LAtwzAMswhhcWcYhlmE1Kq431PpAVSIejxuPub6gI+5xNSk584wDMOcmlqN3BmGYZhTUHPiPtt6rosFIuolot1EtJOIdpjbWonot0R0yPy/pdLjnA9EdC8RDRPRm5ZteY+RDL5jfu5vENEFlRv56VPgmL9CRP3mZ72TiG6yPPZn5jEfIKJ3VGbU84OIVhLRk0S0l4j2ENFnzO2L9rM+xTEv3GcthKiZfzC6Uh4GsBaAG8AuAJsrPa4yHWsvgPasbX8L4Ivm7S8C+GalxznPY7wKwAUA3pztGAHcBOARAATgUgAvVXr8JTzmrwD4fJ59N5vfcQ+ANeZ3X6v0MZzGMXcCuMC83QjgoHlsi/azPsUxL9hnXWuRu1rPVQiRACDXc60XbgHw7+btfwfw7gqOZd4IIbbDaBFtpdAx3gLgR8LgRQDNRNS5MCMtHQWOuRC3ALhfCBEXQhwF0APjN1BTCCEGhRCvmbdnAOyDsTTnov2sT3HMhSj5Z11r4p5vPddTvWG1jADwGBG9SkS3m9uWCiEGzdsnASytzNDKSqFjXOyf/R2mBXGvxW5bdMdMRN0AzgfwEurks846ZmCBPutaE/d64kohxAUAbgTwKSK6yvqgMK7lFnWpUz0co8l3AawDcB6AQQB/X9nhlAciagDwCwD/RwgxbX1ssX7WeY55wT7rWhP3YtZzXRQIIfrN/4cB/BeMS7QheXlq/j9cuRGWjULHuGg/eyHEkBAiLYTQAfwrMpfji+aYicgFQ+T+QwjxS3Pzov6s8x3zQn7WtSbuxaznWvMQUYCIGuVtANcDeBP2tWp/D8CDlRlhWSl0jNsAfMSspLgUwJTlkr6myfKT3wPjswaMY76ViDxEtAbABgAvL/T45gsREYylOPcJIb5teWjRftaFjnlBP+tKZ5VPIwt9E4zM82EAf1Hp8ZTpGNfCyJzvArBHHieANgCPAzgE4H8AtFZ6rPM8zvtgXJomYXiMtxU6RhiVE3eZn/tuAFsrPf4SHvOPzWN6w/yRd1r2/wvzmA8AuLHS4z/NY74ShuXyBoCd5r+bFvNnfYpjXrDPmmeoMgzDLEJqzZZhGIZhioDFnWEYZhHC4s4wDLMIYXFnGIZZhLC4MwzDLEJY3BmGYRYhLO4MwzCLEBZ3hmGYRcj/Ays3uX0XNEW9AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "\n",
    "class Network(object):\n",
    "    def __init__(self, num_of_weights):\n",
    "        # 随机产生w的初始值\n",
    "        # 为了保持程序每次运行结果的一致性，此处设置固定的随机数种子\n",
    "        #np.random.seed(0)\n",
    "        self.w = np.random.randn(num_of_weights, 1)\n",
    "        self.b = 0.\n",
    "        \n",
    "    def forward(self, x):\n",
    "        z = np.dot(x, self.w) + self.b\n",
    "        return z\n",
    "    \n",
    "    def loss(self, z, y):\n",
    "        error = z - y\n",
    "        num_samples = error.shape[0]\n",
    "        cost = error * error\n",
    "        cost = np.sum(cost) / num_samples\n",
    "        return cost\n",
    "    \n",
    "    def gradient(self, x, y):\n",
    "        z = self.forward(x)\n",
    "        N = x.shape[0]\n",
    "        gradient_w = 1. / N * np.sum((z-y) * x, axis=0)\n",
    "        gradient_w = gradient_w[:, np.newaxis]\n",
    "        gradient_b = 1. / N * np.sum(z-y)\n",
    "        return gradient_w, gradient_b\n",
    "    \n",
    "    def update(self, gradient_w, gradient_b, eta = 0.01):\n",
    "        self.w = self.w - eta * gradient_w\n",
    "        self.b = self.b - eta * gradient_b\n",
    "            \n",
    "                \n",
    "    def train(self, training_data, num_epoches, batch_size=10, eta=0.01):\n",
    "        n = len(training_data)\n",
    "        losses = []\n",
    "        for epoch_id in range(num_epoches):\n",
    "            # 在每轮迭代开始之前，将训练数据的顺序随机的打乱，\n",
    "            # 然后再按每次取batch_size条数据的方式取出\n",
    "            np.random.shuffle(training_data)\n",
    "            # 将训练数据进行拆分，每个mini_batch包含batch_size条的数据\n",
    "            mini_batches = [training_data[k:k+batch_size] for k in range(0, n, batch_size)]\n",
    "            for iter_id, mini_batch in enumerate(mini_batches):\n",
    "                #print(self.w.shape)\n",
    "                #print(self.b)\n",
    "                x = mini_batch[:, :-1]\n",
    "                y = mini_batch[:, -1:]\n",
    "                a = self.forward(x)\n",
    "                loss = self.loss(a, y)\n",
    "                gradient_w, gradient_b = self.gradient(x, y)\n",
    "                self.update(gradient_w, gradient_b, eta)\n",
    "                losses.append(loss)\n",
    "                print('Epoch {:3d} / iter {:3d}, loss = {:.4f}'.\n",
    "                                 format(epoch_id, iter_id, loss))\n",
    "        \n",
    "        return losses\n",
    "\n",
    "# 获取数据\n",
    "train_data, test_data = load_data()\n",
    "\n",
    "# 创建网络\n",
    "net = Network(13)\n",
    "# 启动训练\n",
    "losses = net.train(train_data, num_epoches=50, batch_size=100, eta=0.1)\n",
    "\n",
    "# 画出损失函数的变化趋势\n",
    "plot_x = np.arange(len(losses))\n",
    "plot_y = np.array(losses)\n",
    "plt.plot(plot_x, plot_y)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 总结\n",
    "\n",
    "本节，我们详细讲解了如何使用numpy实现梯度下降算法，构建并训练了一个简单的线性模型实现波士顿房价预测，可以总结出，使用神经网络建模房价预测有三个要点：\n",
    "\n",
    "- 构建网络，初始化参数w和b，定义预测和损失函数的计算方法。\n",
    "\n",
    "- 随机选择初始点，建立梯度的计算方法，和参数更新方式。\n",
    "\n",
    "- 从总的数据集中抽取部分数据作为一个mini_batch，计算梯度并更新参数，不断迭代直到损失函数几乎不再下降。"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 1.6.0 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
